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1.0  INTRODUCTION 


Goal:  The  goal  of  this  research  is  to  validate  that  the  Structure 
of  Intellect  (SOI)  model  (Guilford,  1985)  can  be  used  as  a  sufficient 
multi-dimensional  domain  description  for  representing  and  describing 
both  the  capabilities  of  expert  systems  and  candidate  application  domain 
features  and  requirements.  This  representation  will  permit  the 
development  of  an  automated  domain  suitability  evaluation  tool. 

Objective:  The  objective  of  this  research  is  to  develop  and 

validate  an  expert  system  domain  suitability  evaluation  tool.  This 
tool/decision  aid  will  systematically  solicit  information  from  domain 
experts.  The  tool  will  then  use  linear  and  non-linear  multi-dimensional 
analysis  techniques  to  develop  information  on  the  correspondence  between 
the  candidate  domain  and  expert  systems  capabilities  as  they  are 
represented  by  the  Structure  of  Intellect  model.  The  results  of  the 
analysis  should  be  presented  in  a  format  suitable  for  direct 
presentation  to  project  managers  and  expert  systems  designers. 

Problem  statement:  There  are  no  theoretically  based,  empirically 
established  methods  for  analyzing  domain  suitability  for  applications  of 
expert  systems  technology.  In  many  cases,  the  use  of  expert  systems  is 
characterized  by  a  complex  and  costly  development  effort.  Because  of 
the  lack  of  a  useful  domain  evaluation  tool,  systems  designers  must 
first  construct  a  system  before  an  evaluation  can  take  place.  There  is 
a  need  within  the  theory  and  practice  of  expert  systems  implement  .tion 
to  define  and  verify  a  theory-based  methodology  and  analytic  procedures 
for  analyzing  and  evaluating  the  suitability  cf  candidate  domains.  This 
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research  will  contribute  to  answering  this  need  in  the  following  ways. 
First,  this  project  will  serve  to  verify  the  usability  of  the  Domain 
Suitability  Analysis  Tool  ( DSAT )  as  a  comprehensive,  reliable  descriptor 
of  domains.  Second,  this  project  will  define  and  validate  specific 
analytic  i  athods  which  will  define  the  domain  suitability  in  terms  of 
comparisons  with  expert  systems  capabilities.  Last,  a  presentation 
format  will  be  developed  tnat  will  allow  expert  systems  designers  to 
directly  assess  their  candidate  domain  and  receive  a  meaningful 
evaluation  which  is  neither  oversimplified  nor  incomplete. 

The  project  proceeded  from  the  original  development  of  the  DSAT 
model.  From  th: s  starting  point,  v.e  defined  a  problem  statement  and 
objectives  for  the  project.  Figure  1  shows  a  flow  chart  of  how  the 
project  was  performed.  As  can  be  seen  from  the  figure,  the  thrust  of 
the  project  was  in  two  areas.  The  first  was  the  development  of  a 
capability  model  of  current  expert  systems.  This  included  a  review  and 
summary  of  current  documentation  on  expert  systems  and  a  survey  of 
experts  in  the  area.  The  second  effort  was  to  develop  and  validate  the 
numerical  lata  analysis  methodology. 

The  development  of  this  new  methodology  marks  a  fundamental  change 
in  the  way  domain  selection  is  done.  The  existing  methods  for  selecting 
domains  a. id  evaluating  their  suitability  consists  of  doing  subjective 
comparisons  of  the  new  domain  with  past  development  efforts.  This  new 
methodology  starts  with  a  compact  parametric  description  of  the  domain 
and  a  comprehensive  description  of  the  capabilities  of  expert  systems. 
From  these,  the  analysis  compares  requirements  and  capabilities  to 
determine  both  the  suitability  and  predict  the  problem  areas  in  a 
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development  effort.  The  differences  between  the  old  and  new  method  for 


domain  description  are  shown  in  Figure  2. 

There  has  been  work  in  the  last  few  years  to  develop  methods  for 
evaluating  domain  suitability.  The  majority  of  the  efforts  have 
concentrated  on  developing  subjective  cost  benefit  analyses  by  breaking 
out  and  evaluating  various  aspects  of  the  development  of  an  expert 
system  for  a  specific  domain  application.  Jay  Horn,  in  his  work  at 
Wright  State,  developed  a  concept  of  detailed  analysis  of  candidate 
domains  to  determine  the  practically  of  the  expert  system  development 
effort. 

The  DSAT  was  developed  to  address  a  comprehensive  evaluation  of 
candidate  domains  for  expert  systems  projects  in  a  theoretically  based 
deterministic  comparison  with  capabilities  inherent  in  expert  systems. 
However,  his  efforts  did  not  include  construction  of  more  than  a  very 
simple  metric  of  the  data  analysis. 

Typical  of  the  current  methods  in  use  at  this  time  is  the  "tarot 
metric".  This  method  defines  a  decision  rule  whose  output  is  whether  to 
go  ahead  with  an  expert  system  development  effort.  Evaluation  factors 
for  this  method  are  listed  below. 

Evaluation  Factors:  Worth 

Risk 

Employee  Acceptance 
Solution  Availability 
Easier  Solution 
Teachability 

These  factors  are  used  as  part  of  a  weighted  decision  rule.  The 
system  designer  makes  a  subjective  evaluation  of  each  factor  and  then 
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decides  whether  to  go  ahead  with  the  development  based  on  the  weighted 
sum  of  the  factors  in  the  decision  rule. 

When  we  look  at  an  evaluation  of  the  suitability  of  a  domain  for 
the  application  of  expert  systems,  it  becomes  clear  that  the  problem  of 
describing  the  suitability  is  a  function  of  the  description  of  the 
domain  itself,  as  it  compares  to  the  capability  of  expert  systems 
technology.  Therefore  it  is  not  effective  to  generate  a  single  value  for 
such  a  complex  multi-dimensional  problem. 

The  purpose  of  the  data  analysis  is  the  development  of  an 
empirically  established  algorithm  for  producing  a  domain  suitability 
index  that  represents  the  capabilities  and  limitations  of  expert  systems 
technology  and  domain  characteristics  relative  to  the  elements  specified 
in  the  SOI  model  (see  Figure  3).  Figure  3  shows  the  structure  of  the 
Intellect  Cube.  The  cube  shows  graphically  how  the  three  dimensions  of 
the  SOI  model  combine  to  represent  all  elements  of  intelligent  tasks. 

The  principal  dimensions  of  the  SOI  model  are  contents,  operators,  and 
products.  Data  will  be  collected  for  the  analysis  using  the  data 
acquisition  tool  developed  by  Jay  Horn.  He  has  demonstrated  that  the 
DSAT  format  will  consistently  gather  data  regarding  candidate  domains  to 
support  the  Guilford  Structure  of  Intellect  model.  The  mapping  system 
of  analysis  is  based  on  two  basic  constructs.  The  first  of  these  is 
that  the  Structure  of  Intellect  model  of  a  domain  and  expert  systems 
capability  can  be  represented  as  a  plot  or  profile  of  various  attributes 
defined  within  the  model.  The  second  is  that  we  can  compare  the 
individual  elements  of  the  SOI  model  on  a  one  to  one  basis  to  determine 
the  overlaps  and  shortcomings  of  expert  systems  technologies  in  relation 
to  the  domain  of  interest.  The  first  issue  has  been  defined  and 
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verified  by  Jay  Horn's  research  and  will  be  further  verified  during  this 
project.  The  latter  issue  has  been  verified  a  number  of  times  in 
dealing  with  cognitive  abilities  in  a  multi-channel  system.  Notably  in 
a  study  by  Wickens  (1988)  in  which  he  predicts  system  performance  based 
on  matching  profiles  of  needed  and  available  attributes  for  decision 
making  (see  Figure  4).  In  Figure  4,  we  see  how  Wickens  predicts 
performance  by  representing  capability  in  terms  of  a  multi-dimensional 
plot,  and  the  comparison  of  the  demands  of  a  particular  task  defined  in 
terms  of  the  same  multi-dimensional  plot.  The  SOI  model  domain  will  be 
defined  by  a  set  of  scaled  vectors,  each  representing  a  dimension  in  the 
model.  The  set  of  these  vectors  will  define  a  N-dimensional  vector 
space  into  which  each  domain  will  define  a  topology  based  on  the  values 
for  each  parameter  of  the  domain. 

To  make  the  mapping  analysis  feasible,  a  map  of  expert  systems 
attributes  in  the  same  format  as  the  domain  representations  is  required. 
This  map  will  be  developed  by  interviewing  various  knowledge  engineers 
on  the  nature  and  abilities  of  current  expert  systems  and  constructing  a 
model  of  expert  systems  capabilities  in  the  same  form  as  the  DSAT  domain 
representation.  During  the  interviews  with  the  knowledge  engineers,  the 
strengths  and  weaknesses  of  expert  systems  will  be  determined.  In 
addition  to  this,  we  will  try  to  learn  something  about  the  flexibility 
of  system  implementations  and  what  would  be  a  useful  and/or  acceptable 
format  to  present  domain  information  to  the  design  engineers. 

Once  the  domain  of  interest  and  the  expert  system  capabilities  are 
mapped  in  the  same  space,  we  can  then  make  direct  comparisons  between 
them.  The  comparison  will  be  grouped  into  two  categories:  1)  the 
general  shape  or  global  analysis  of  the  two  maps,  and  2)  a  point-by- 
point  comparison  of  the  maps  to  determine  the  dimensions  the  DSAT  model 
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in  which  the  expert  systems  capabilities  may  not  support  the  demands  of 
the  domain.  The  mapping  comparisons  will  highlight  critical  elements 
which  will  be  studied  in  the  second  phase  of  the  analysis.  An  example 
of  global  analysis  methods  is  a  comparison  of  the  number  of  elements  of 
the  model  which  are  active  in  a  given  domain  or  how  these  elements  are 
arranged  with  regards  to  one  another.  An  example  of  the  point-by-point 
comparison  is  to  compare  the  values  of  a  single  element  from  one  domain 
to  another. 

Critical  Element  Analysis 

The  critical  element  is  a  secondary  interview  to  obtain  more 
information  about  the  most  important  elements  of  the  domain  as 
determined  by  the  mapping  of  the  data  collected  by  the  DSAT  tool.  To 
determine  which  elements  to  examine  further,  we  will  develop  a  threshold 
function  which  will  be  a  function  of  the  raw  data  from  the  original  DSAT 
data  and  the  mapping  analysis.  This  will  allow  us  to  look  at  the  most 
important  elements  of  each  domain  independent  of  predetermined 
procedures.  The  threshold  function  will  be  developed  based  on  such 
factors  as  overall  value  of  the  DSAT  raw  data,  differences  between  the 
domain  and  the  expert  system,  and  uncommon  factors  in  the  data. 

In  order  to  create  a  usable  tool  for  the  computer/expert  systems 
industry,  the  data  collection  and  analysis  will  be  automated  and 
implemented  on  a  personal  computer  format.  An  automated  software 
product  will  evaluate  and  map  a  prospective  domain  onto  the  expert 
system  capability  map,  and  analyze  key  elements  of  the  critical 
attributes  of  the  domain. 

When  looking  at  whether  the  model  is  comprehensive  and  valid,  we 
need  to  look  at  describing  some  of  the  other  expert  systems  evaluation 
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factors  in  terms  of  the  Structure  of  Intellect  model.  The  development 
of  the  analytic  methods  will  start  by  using  data  collected  by  Jay  Horn. 
Using  this  data,  we  will  be  able  to  construct  mappings  based  on  actual 
domain  representations.  Because  the  critical  element  analysis  can  be 
done  after  the  original  DSAT  analysis,  this  analysis  can  be  done  on  the 
domains  previously  documented  by  Jay  Horn.  To  enrich  the  overall  data 
base,  additional  data  will  also  be  collected  regarding  additional 
domains.  In  selecting  additional  domains  for  data  collection,  domains 
were  chosen  which  would  increase  the  coverage  of  the  SOI  modeling  space 
and  for  which  some  data  exists  on  how  well  expert  systems  development 
efforts  have  gone  in  the  past.  This  better  enables  us  to  draw 
conclusions  from  this  data. 
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The  Structure  of  intellect  Cube  (Guilford,  1967;  1985). 
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Profile  of  pilot's  cognitive  attributes,  along  wich  cwo 
representative  scenario  profiles.  To  the  extent  that  the  demands  of 
the  scenerio  match  the  pilot's  profile,  good  performance  is  predicted. 
To  the  extent  that  a  mismatch  occurs,  poorer  performance  is  expected. 
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2.0  THEORY  AND  BACKGROUND 


2.1  THEORETICAL  BASIS 

2.1.1  MOTIVATION 

Jay  Horn  demonstrated  that  the  DSAT  tool  can  be  of  considerable 
use  to  the  research  community,  as  well  as  to  decision  makers  in  the 
development  of  practical  expert  systems.  By  development  of  more 
complete  and  descriptive  analysis  procedures  the  DSAT  tool  will  be  able 
to  give  the  system  decision  makers  and  the  system  design  engineers  a 
descriptive  view  for  the  domain  they  will  be  working  with  and  hopefully 
some  specific  information  as  to  the  interaction  between  the  domain  and 
the  human  experts  which  interact  with  it.  With  the  development  of  the 
critical  element  analysis  procedures,  it  will  be  possible  to  give 
information  to  the  designers  as  to  the  relative  importance  and/or 
interaction  between  the  domain  elements. 

2.1.2  PSYCHOMETRIC  THEORIES  OF  INTELLIGENCE 

The  psychometric  theorists  use  individual  differences  data  (refined 
through  the  use  of  factor  analysis)  to  separate  patterns  of  reasoning 
from  other  abilities  and  to  examine  the  various  reasoning  skills 
exhibited.  These  various  methods  have  relied  on  individual  difference 
data  for  testing  and  formulating  theories  (Guilford  and  Hoepfner,  1971). 

Spearman  (1904),  who  refined  the  factor  analysis  method,  is 
regarded  as  one  of  the  first  psychometric  theorists.  He  developed  a 
factor  theory  of  intelligence  (specifically  for  intelligence  testing) 
which  proposed  a  single,  central  element  of  intelligence,  G  (Spearman, 
1904).  The  second  factor  was  originally  believed  to  be  a  factor  of 
intelligence  unique  to  specific  tests,  and  would  be  indiscernible  by  any 
other  test. 
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However,  as  results  of  factor  analysis  mounted,  there  appeared  to 
be  other  related  factors  that  inter-correlated  more  strongly  than 
initially  predicted  by  the  theory.  Spearman  called  variables  common  to 
a  specific  group  of  tests  "group  factors."  He  attempted  to  downplay  the 
importance  of  these  factors,  emphasizing  the  importance  of  G  instead 
(Guilford  and  Hoepfner,  1971). 

Thurstone  (1938),  intrigued  by  the  existence  of  group  factors,  used 
over  50  different  intelligence  tests  to  uncover  a  number  of  intellectual 
factors  in  a  large  group  of  college  subjects.  He  proposed  a  theory  of 
primary  mental  abilities  (PMA)  composed  of  seven  factors.  These  were 
verbal  comprehension,  numerical  facility,  spatial  ability,  perceptual 
speed,  rote  memory,  induction,  and  deduction. 

The  next  step  in  the  growth  of  psychometric  theories  of 
intelligence  came  during  World  War  II.  J.  P.  Guilford,  as  director  of 
Psychology  Unit  #3  of  the  Aviation  Psychology  Research  Program,  was 
asked  to  determine  selection  criteria  for  aircrew  personnel  in  the 
intellectual  area.  Examination  of  the  reasons  students  washed-out  of 
pilot  training  revealed  eight  general  psychological  constructs: 
judgment,  foresight  and  planning,  memory,  comprehension,  visualization 
of  flight  path,  spatial  orientation,  reasoning,  and  coordination  of 
information.  Guilford  and  Lacey  (1947)  performed  an  in-depth  factor 
analysis  of  these  constructs  and  demonstrated  approximately  25 
intellectual  factors. 

After  World  War  II,  Guilford  continued  investigation  of  these 
factors  of  intelligence  under  the  aegis  of  the  Aptitudes  Research 
Project  for  the  Office  of  Naval  Research.  The  early  years  of  the 
project  demonstrated  all  of  Thurstone' s  primary  mental  abilities,  as 
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well  as  finding  two  aspects  of  his  spatial  factor;  arrangement  of 
objects  in  space,  and  visualizing  changes  in  objects  (Guilford,  1985). 
This  brought  the  list  of  abilities  to  near  40. 

As  the  list  of  factors  began  to  mount,  several  similarities  and 
differences  became  apparent.  Some  factors  could  be  grouped  based  on  the 
mental  processes  involved,  such  as  cognition,  memory,  and  evaluation. 
Others  could  be  segregated  based  on  the  information  used;  symbolic, 
semantic,  or  visual,  for  example.  A  third  dimension  involved  the  form 
of  the  information  used;  units,  classes,  or  relations,  for  example. 

The  resulting  cubic  figure  which  encompasses  these  dimensions  is 
the  Structure  of  Intellect  (SOI)  model  (Guilford,  1967;  Guilford,  1985). 
Figure  3  shows  the  SOI  model  with  its  three  dimensions:  operations, 
contents  and  products.  This  model  illustrates  the  focus  of  psychometric 
theories  on  cataloging  and  systematically  identifying  the  components  of 
intelligence  and  cognitive  behavior. 

2.1.3  STRUCTURE  OF  INTELLECT  MODEL 

Guilford's  SOI  model  is  the  most  logical  choice  as  a  theoretical 
basis  for  examining  domain  suitability  with  respect  to  rule-based  expert 
systems  technology.  To  summarize,  reasons  for  selecting  this  model  over 
others  are:  this  model  has  sensitivity  to  many  aspects  of  a  domain  and 
unambiguous  representation  of  those  aspects  (over  150  attributes  can  be 
specified);  the  specificity  of  the  SOI  model  will  enhance  the 
reliability  of  the  tool;  the  SOI  model  has  high  content  validity;  and 
the  SOI  model  has  been  developed  over  the  past  40  years,  resulting  in 
several  refinements  and  an  "accumulation  of  empirical  evidence" 
suggesting  its  construct  validity  for  this  approach. 
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Content  Validation 


The  SOI  model  meets  this  criterion  since  the  model  examines  over 
150  facets  of  a  domain.  Specification  of  a  particular  operator, 
content,  and  product  specifies  the  type  of  process,  the  type  of 
information  used,  and  the  type  of  information  produced  during  the 
activity.  This  model  allows  sufficient  domain  coverage  to  establish  a 
high  degree  of  satisfactory  content  validity. 

Criterion-Related  Validity 

Both  predictive  and  concurrent  validities  have  been  established  for 
the  SOI  model  in  many  different  intelligence  and  cognition  related 
applications  such  as  intelligence  testing,  creativity  measurement,  and 
job  selection  (Guilford,  1985;  Guilford  and  Hoepfner,  1971;  Meeker, 
1969).  While  the  theory  does  possess  satisfactory  levels  of  criterion- 
related  validity,  the  specific  criterion-related  validity  for  this 
application  must  be  established. 

Construct  Validity 

While  the  theory  has  existed  for  over  three  decades,  it  remains  a 
viable  tool  for  understanding  the  various  aspects  of  human  intelligence. 
Kolodner  (1964)  posits  that  the  structure  of  information  used  in  human 
intellectual  problem  solving  is  different  in  experts  than  in  novices, 
based  on  her  research.  The  importance  of  information  structure  in  the 
mind  of  the  problem  solver  (cited  by  Kolodner)  suggests  that  a 
structural  approach,  like  the  SOI  model,  is  useful  in  describing 
application  domains.  This  lends  some  support  to  the  existence  of 
construct  validity  in  this  theory. 
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Reliability 


The  layout  of  the  SOI  model  and  the  terms  used  lend  themselves  to  a 
reliable  method  of  describing  application  domains  and  expert  systems 
technology  attributes.  The  development  of  logical  and  specific 
administration  procedures  will  further  insure  the  reliability  of  the 
tool.  Jay  Horn's  development  demonstrated  the  reliability  of  the  DSAT 
instrument . 

2.1.4  ANALYSIS  METHODS 

The  DSAT  is  an  inherently  multi-dimensional  representation  of  the 
cognitive  and  signal  processing  structure  of  complex  task  domains. 
Therefore,  it  is  reasonable  to  derive  a  multi-dimensional  model  of  the 
domain  characteristics  which  are  elicited  by  the  DSAT  questionnaire. 

The  multi-dimensional  model  will  be  used  to  describe  the  domain  as 
a  whole  and  derive  global  characteristics  for  the  domain.  However,  this 
form  of  analysis  is  not  well  suited  to  deriving  specific  relationships 
between  the  key  factors  or  elements  that  make  up  any  given  domain.  In 
fact,  the  basic  DSAT  questionnaire  is  not  well  suited  to  this  analysis. 
It  is  kno  m  from  previous  research  (Horn,  1988)  that  human  experts  can 
make  very  fine  discriminations  between  factors  if  comparing  them  side  by 
side.  We  can  use  this  ability  to  gain  a  great  deal  of  information  about 
the  key  factors  which  contribute  most  to  definin  the  nature  of  a  given 
domain  of  interest.  Therefore,  the  maximum  possible  information  about 
these  domains  can  he  obtained  by  examin^the  spanning  set  of  their 
parameters . 

Excursion  Analysis 

Excursion  analysis  allows  the  analysis  of  the  data  describing  the 
domain  of  interest  to  concentrate  on  those  areas  which  are  not  well 
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implemented  by  expert  systems.  Wickens,  in  his  studies  of  workload 
(1987),  used  the  technique  of  excursion  analysis  to  describe  what  tasks 
a  human  operator  might  reasonably  be  expected  to  perform  based  on  a 
description  of  the  task  and  the  difference  between  task  demands  and  the 
capabilities  of  the  operator.  In  excursion  analysis,  the  object  is  to 
map  the  two  items  of  interest  (which  in  this  case  are  expert  systems 
capabilities  and  domain  requirements)  in  the  same  coordinate  system  so 
that  they  can  be  compared  on  a  one  to  one  basis.  Therefore,  the 
challenge  becomes  to  develop  an  equivalent  representation  for  expert 
systems  capabilities  and  domain  requirements. 

Critical  Incident  Analysis 

Critical  incident  or  critical  element  analysis  is  widely  used  for 
identifying  key  elements  in  a  larger  system  or  representation.  There 
are  a  number  of  studies  and  principles  which  demonstrate  that  the 
majority  of  problems  can  be  represented  by  only  a  few  of  the  most 
important  subgroups  (Perado  Principle).  The  excursion  analysis  allows 
for  a  first  level  filter  of  the  elements  of  the  SOI  model  that  are  of 
the  most  importance.  Additional  thresholding  algorithms  for  critical 
incident  analysis,  based  on  difficulty  and  criticality,  are  developed  in 
section  4.2. 

Neural  Net  Theory  and  Explanation  Application 

Artificial  neural  net  models  or  simply  "neural  nets"  go  by  many 
names  such  as  connectionist  models,  parallel  distributed  processing 
models,  and  neuromorphic  systems.  Whatever  the  name,  all  of  these 
models  attempt  to  achieve  good  performance  via  dense  inter-connection  of 
simple  computational  elements.  In  this  respect,  artificial  neural  net 
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structure  is  based  on  our  present  understanding  of  biological  systems. 
Computational  elements  or  nodes  used  in  neural  net  models  are  nonlinear 
and  are  typically  analog.  The  simplest  node  sums  N  weighted  inputs  and 
passes  the  results  through  a  nonlinearity.  Neural  net  models  are 
specified  by  the  net  topology,  node  characteristics,  and  training  or 
learning  rules.  These  rules  specify  an  initial  set  of  weights  and 
indicate  how  weights  should  be  adapted  during  use  to  improve 
performance. 

Traditional  neural  nets  are  good  as  classifiers.  Classifiers  can 
perform  three  different  tasks.  First,  they  can  identify  which  class 
best  represents  an  input  pattern.  Second,  the  classifiers  can  be  used 
as  a  content-addressable  or  associative  memory,  where  the  class  exemplar 
is  desired  and  the  input  pattern  is  used  to  determine  which  exemplar  to 
produce.  A  third  task  these  classifiers  can  perform  is  to  quantify 
vectors  or  cluster  the  N  inputs  into  M  clusters  (Lippmann,  1987). 

The  problem  of  designing  a  system  tool  for  domain  evaluation  is 
well-suited  to  the  application  of  neural  net  classifiers.  The  domain 
description  from  the  DSAT  tool  will  serve  as  the  input  to  the  classifier 
and  the  desired  output  of  the  classifier  will  be  a  numeric  evaluation  of 
whether  or  not  the  domain  falls  in  a  class  of  domains  well-suited  to  an 
expert  systems  implementation. 
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2 . 2  BACKGROUND 


This  section  reviews  the  current  status  of  expert  systems  domain 
evaluation  and  selection. 

The  procedures  necessary  to  perform  an  accurate  evaluation  of 
potential  application  domains  for  expert  systems  technology  using  the 
theory-based  model  have  been  integrated  to  create  the  Domain  Suitability 
Analysis  Tool  (DSAT) .  The  DSAT  has  been  designed  to  be  both  theoreti¬ 
cally  valid  and  operationally  efficient.  Validation  of  the  DSAT  was 
achieved  by  examining  several  diverse  application  domains  (to  which 
expert  systems  suitability  had  previously  been  established)  and 
comparing  the  recommendations  of  the  DSAT  against  the  existing  knowledge 
of  the  domain's  suitability. 

The  DSAT  was  designed  to  describe  the  domain  information  elements 
(based  on  the  SOI  model)  as  revealed  by  the  domain  expert  and  produce  a 
recommendation  based  on  the  degree  of  support  provided  by  expert  systems 
technology  for  those  information  elements.  While  some  researchers  have 
attempted  (with  little  success)  to  map  domains  in  terms  of  expert 
systems  capabilities,  we  define  expert  systems  suitability  in  terms  of 
the  domain's  information  requirements. 

Jay  Horn's  work  demonstrates  that  expert  subjects  produced  very 
good  reliability  in  inter-subject  data  convergence  both  on  an  absolute 
basis  and  relative  to  novice  subjects,  expert  (0.305)  and  novice 
(1.075).  Therefore,  we  chose  not  to  collect  data  from  novice  subjects 
and  will  concentrate  on  experts  from  differing  domains,  in  an  effort  to 
increase  the  convergence  and  understanding  of  the  performance  of  the 
DSAT  tool. 
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2.2.1  PROBLEM  STATEMENT 


There  are  no  theoretically-based,  empirically  established  methods 
for  determining  domain  suitability  for  applying  rule-based  expert 
systems  technology.  While  several  ad  hoc  guidelines  have  been  developed 
and  are  currently  in  use,  they  provide,  at  best,  only  general  guidelines 
for  determining  if  a  domain  can  benefit  from  rule-based  implementations 
of  expert  systems  technology.  Hadzikadic,  Yun,  and  Ho  (1987)  succinctly 
state  the  problem: 

"As  expert  systems  technology  becomes  increasingly 
popular  for  (yet)  untested  applications,  a  serious 
gap  in  knowledge  has  become  increasingly  prominent. . 

. .the  appropriateness  of  the  match  between  a 
prospective  application  domain  and  the  tools  of  ES 
(expert  systems)."  (p.64) 


Kidd  and  Sharpe  (1988)  are  also  concerned  with  the  lack  of 
theoretically-based  research  which  accounts  for  human  expertise  in 
artificial  intelligence.  They  regard  the  current  generation  of  expert 
systems  as  experiments,  which  have  focused  impressive  amounts  of 
computational  power  on  specific  problems  in  highly  isolated  domains,  but 
have  yet  to  achieve  sufficient  basis  in  theory  to  allow  growth  into 
diverging  domains.  They  write: 


"Success  of  the  system  is  directly  determined  by 
the  appropriate  representation  and  application  of 
specific  knowledge  from  that  domain  to  solve  an 
isolated  problem.  Despite  the  vast  amount  of  data 
now  available  as  a  result  of  these  experiments, 
we  are  still  unable  to  explain  the  "why"  or  "how" 
of  successful  systems  or  to  predict  for  which  other 
domains  and  tasks  the  current  techniques  will  work. 

This  is  because  no  theory  of  tasks  or  domains 
currently  exists."  (p.147). 

The  taxonomy  of  the  domain  information  structure  that  is  specified 
by  this  research  can  be  useful  in  identifying  critical  areas  of  a  domain 
which  deserve  additional  attention  during  the  expert  systems  design 
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process.  The  taxonomy  can  also  be  used  to  estimate  the  allocation  of 
functions  between  the  operator  and  the  machine,  based  upon  the  structure 
and  processing  required  of  specified  information  elements. 

2.2.2  EVAI  UATION  OF  THE  DSAT  AS  A  DATA  ACQUISITION  TOOL 

The  nature  of  the  SOI  model  lends  itself  to  unambiguous  definition 
of  various  attributes  under  consideration  in  a  domain.  The  vast 
majority  of  all  attributes  composing  the  SOI  model  have  been 
operationally  defined,  significantly  reducing  the  possibility  of 
misidentifying  an  attribute  (Guilford,  1987).  This  characteristic  is 
also  useful  in  establishing  the  sensitivity  of  the  theory.  The 
significant  number  of  attributes  contributes  to  the  ability  to  specify 
differences  between  domains. 

The  DSAT  questionnaire  is  structured  to  collect  data  about  each  of 
the  sections  of  the  Guilford  Structure  of  Intellect  Cube.  For  each 
section  of  the  cube,  the  questionnaire  obtains  a  rating  for  Frequency, 
Difficulty,  and  r  iticality.  The  DSAT  uses  a  10-point  scale  for 
comparing  ratings  across  dimensions.  Jay  Horn  (1989)  reported  on  the 
reliability  of  the  DSAT  questionnaire  (see  Jay  Horn's  thesis).  His 
results  suggest  that  if  properly  used  by  a  skilled  knowledge  engineer  on 
highly  experienced  domain  experts,  the  DSAT  questionnaire  is  both 
reliable  and  sensitive  in  defining  the  domain  representations. 

Organization  of  the  DSAT 

The  DSAT  questionnaire  involves  a  series  of  questions  about  the 
domain  of  interest  regarding  the  various  dimensions  of  the  SOI  model. 

The  DSAT  has  two  sections:  Part  I  examines  the  structural  components 
based  on  the  content-product  aspects  of  the  domain;  Part  II  examines  the 
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operational  components  which  incorporate  the  domain  operators  with 
element  frequency,  criticality,  and  difficulty. 

The  first  phase  assesses  the  structural  components  of  the  domain. 
By  using  the  content-product  dimension  of  the  SOI,  a  30-element 
psychoepistemology  is  described  which  provides  a  means  for  determining 
the  basic  informational  elements  evident  in  the  domain.  By  having  the 
domain  expert  indicate  the  presence  or  absence  of  each  component,  the 
structural  information  requirements  for  the  domain  are  established. 

The  second  phase  of  the  DSAT  administration  involves  determining 
the  relevant  operators  (mental  function)  required  to  act  upon  the 
information  elements  defined  in  the  previous  step,  and  to  establish  the 
criticality,  frequency,  and  difficulty  of  the  various  components  to 
allow  comparison  with  expert  systems  technology  attributes.  As  an 
example,  assume  10  of  the  30  content-product  elements  are  identified  as 
domain-relevant  in  Part  I.  In  Part  II,  the  domain  expert  identifies 
which  operators  are  relevant  for  each  of  the  10  elements  and  the  degree 
of  frequency,  criticality,  and  difficulty  associated  with  each. 

The  evaluation  metric  which  Jay  Horn  used  in  his  original  study  of 
the  DSAT  tool  was  a  normalized  linear  weighted  combination  of  the  values 
obtained  from  the  raw  survey.  The  combination  algorithm  was  Capt  Horn’s 
best  estimation  of  the  relative  contributions  of  the  DSAT  dimensions  to 
the  suitability  of  the  domain  to  expert  systems  solution.  The  algorithm 
from  his  thesis  is  stated  again  here  to  form  a  basis  for  the  reader  to 
consider  as  we  develop  the  complete  model. 
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OCR  =  Operator  Combined  Rating 
OCR  =  0.10  F  +  0.35  C  +  0.50  D  +  0.05  N 
F  =  Frequency  rating 
C  =  Criticality  rating 
D  =  Difficulty  rating 
N  =  Number  of  information  elements 

Domain  Suitability  Index  (  DSI  ) 

DSI  =  0.25  (  C  OCR  )  +  0.05  (  M  OCR  )  +0.5  (  DP  OCR  ) 

+  0.05  (  CP  OCR  )  +  0.15  (  E  OCR  ) 

C  =  Cognition  operator 
DP  =  Divergent  Production 
E  =  Evaluation  operator 
M  =  Memory 

CP  =  Convergent  Production 

2.2.3  DOMAIN  EVALUATION  TOOLS 

The  early  theoretical  approaches  laid  the  ground  work  for  the 
present  research.  For  example,  the  Additive  Rating  Model  Methodology 
( ARMM)  of  Bringelson,  De  McCray,  Thompson,  and  Salvendy  (1987) 
attempted  to  define  a  method  of  specifying  job  tasks  and  skills  in  terms 
of  expert  systems  capabilities. 

The  task  list  (from  Waterman,  1986)  defines  11  tasks  "involved  in 
knowledge  based  systems."  The  24  skills  involved  in  those  tasks  are 
specified  from  the  work  of  Lenorovitz,  Phillips,  Ardrey,  and  Kloster 
(1984)  which  examined  skills  used  in  human-computer  interactions. 

Table  1  illustrates  this  matrix. 

The  product  of  the  matrix  evaluation  is  a  numerical  rating  of  the 
skills  encompassed  in  a  task  (see  Table  1).  The  scores  can  range  from  0 
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(no  skills  are  replicable)  to  24  (all  skills  are  replicable).  Higher 


scores  imply  more  skills  can  be  replicated  by  the  computer,  better 
defining  the  task  and  resulting  in  better  task  performance.  The  scores 
for  each  task  of  a  job  are  then  summed  and  divided  by  the  number  of 
tasks  to  obtain  a  composite  score.  Bringelson,  et  al.  (1987)  define 
scores  above  17  (one  standard  deviation  above  the  mean  task  score  of 
11.73)  as  "good"  (expert  systems  are  applicable),  scores  12  to  17  as 
"marginal"  (expert  systems  may  be  applicable) ,  and  scores  below  12  as 
"poor"  (expert  systems  do  not  apply) . 

In  effect,  the  tool  confirms  much  of  what  we  already  know.  Unfor¬ 
tunately,  the  ambiguous  job  domains  are  the  most  prevalent,  and  this  is 
where  the  tool  must  be  accurate.  While  the  overall  concept  of  the 
Additive  Bating  Model  Methodology  (ARMM)  is  laudable,  there  are  several 
problems  that  encourage  additional  research.  The  difficulties  of 
achieving  a  high  level  of  validity  and  reliability,  identifying  relevant 
information  types  and  processes  used  in  the  job,  and  achieving  sensitiv¬ 
ity  among  complex  domains  all  point  toward  the  need  for  a  theoretically- 
based  approach  that  can  moderate  these  deficiencies  and  provide  useful 
direction  for  domain  assessment. 

Guidelines  For  Domain  Selection 

While  each  of  the  researchers  cited  by  Allen  (1986)  identifies  the 
selection  of  a  suitable  problem  domain  as  an  important  step  in  the 
development  process,  none  go  farther  than  suggesting  a  few  broad  guide¬ 
lines  for  making  this  selection.  Several  ad  hoc  guidelines  for  domain 
selection  have  been  proposed.  Table  2  lists  several  of  the  observations 
cited  by  Prerau  (1985)  and  Dreyfus  and  Dreyfus  (1986)  as  typical  of 
these  guidelines. 
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The  major  objection  to  these  guidelines  is  that  they  are  based  on  ad 
hoc  observations  of  past  successful  cases.  Dreyfus  (1979)  quotes  AI 
researcher  Drew  McDermott,  "...  AI  (and  expert  systems)  is  a  field 
starving  for  a  few  carefully  documented  failures"  (p.  46).  By  only 
examining  the  "successes"  of  any  endeavor,  useful  information  and 
"lessons  learned"  are  lost.  The  conclusions  drawn  from  this  kind  of 
analysis  are  subject  to  error  and  tend  to  be  incomplete. 

A  second  problem  with  these  guidelines  is  the  lack  of  clear 
recommendations  regarding  the  utility  of  expert  systems  technology  for  a 
specific  problem.  Use  of  general  guidelines  places  the  decision-maker 
in  the  position  of  making  potentially  cost-intensive  decisions  based  on 
largely  subjective  estimates  using  incomplete  criteria. 

The  current  guidelines  do  not  provide  specific  guidance  regarding 
functional  allocation  of  duties  between  the  user  and  the  system.  The 
theoretically-based  methodology  proposed  here  attempts  to  produce  a 
model  of  domain  and  expert  system  attributes  that  can  be  used  to  examine 
the  relative  strengths  and  weaknesses  of  each  domain  as  it  compares  to 
the  capabilities  of  expert  systems. 

As  problems  and  domains  evolve  over  time,  they  change  in  many  ways. 
The  advantage  of  the  ESDAAT  is  that  we  capture  the  fundamental  priori¬ 
ties  of  the  domain  and  not  just  the  parameters  of  a  particular  solution 
to  the  domain  problems.  Therefore,  the  ESDAAT  is  be  much  more  tolerant 
to  changes  in  the  domain  solutions  than  other  evaluation  tools.  The 
ESDAAT  will  differ  significantly  from  other  expert  system  advisors 
because  it  performs  its  analysis  based  on  a  concise  definition  of  expert 
systems  capabilities. 
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TABLE  1 


Skills,  Tasks,  and  Scores  Used  in  the  Additive  Rating  Model  Methodology 


Skills  that  can  be  emulated  using  current  expert  systems 
technology  (Lenorovitz,  Phillips,  Ardrey,  and  Kloster,  1984.) 


Detect 

Search 

Scan 

Extract 

Cross-Reference 

Recognize 

Categorize 

Calculate 

Itemize 

Discriminate 

Tabulate 

Estimate 

Translate 

Compare 

Interpolate 

Formulate 

Integrate 

Evaluate 

Select 

Extrapolate 

Acknowledge/ 

Respond 

Direct/ 

Inform 

Ungroup/ 

Segregate 

Filter 

Tasks  performed  in  a  job  domain  (Waterman,  1986),  their  associated 
scores  of  skill  components,  and  qualitative  suitability  (with 
appropriate  score  range)  for  expert  systems  applications. 


GOOD 

MARGINAL 

POOR 

(17.00  -  24.00) 

(11.73  -  16 

.99) 

(0.00  -  11.72) 

Diagnosis 

-23 

Instruction 

-15 

Design  -11 

Conflict 

resolution 

-23 

Monitoring 
&  gathering 

-11 

Information  -11 

Prediction 

-21 

Creating 

-8 

Planning 

-21 

Interpretation 

-20 

Delegation 

-19 

(Source:  Bringelson,  Deer,  McCray,  Thompson,  and  Salvendy,  1987) 
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TABLE  2 


Guidelines  for  Selecting  an  Appropriate  Expert  Systems 

Application  Domain 

Dreyfus  and  Dreyfus  (1986). 

1.  No  algorithmic  solution  to  the  problem  should  exist. 

2.  The  problem  can  be  satisfactorily  solved  by  human  experts  at  such  a 
high  level  that  somewhat  inferior  performance  is  acceptable. 

3.  Non-experts  have  a  high  probability  of  making  a  poor  decision. 

4.  Poor  decisions  have  significant  impacts. 

5.  The  problem  is  stable  during  the  time  taken  to  make  a  decision. 

6.  The  knowledge  domain  must  be  relatively  static. 

7.  An  expert  must  be  available  to  provide  the  knowledge  base. 


Prerau  (1985).  (This  list  is  a  sample  of  his  52  guidelines.) 

1.  The  domain  is  characterized  by  the  use  of  expert  knowledge. 

2.  Conventional  programming  approaches  to  the  task  are  unsatisfactory. 

3.  There  are  recognized  experts  that  solve  the  problem  daily. 

4.  Experts  are  probably  better  than  amateurs  in  performing  the  task. 

5.  There  is  a  need  to  "capture"  the  expertise  for  the  future. 

6.  The  task  is  neither  too  easy  nor  too  difficult  for  an  expert. 

7.  Domain  selected  offers  the  greatest  return  for  the  projected  risk. 

8.  The  task  primarily  involves  symbolic  reasoning. 

9.  The  task  requires  the  use  of  heuristics  and  may  require  the 
consideration  of  many  alternatives  or  decisions  based  on  incomplete  or 
uncertain  information. 

10.  The  task  inputs  and  outputs  are  clearly  defined  at  the  outset. 
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3.0  STATEMENT  OF  OBJECTIVES 


3 . 1  GOAL 

The  goal  of  this  research  is  to  develop  an  implementation  tool  for 
evaluating  domain  suitability  for  expert  systems  technology  applications 
using  multi-dimensional  and  critical  element  analysis,  and  to  solicit 
and  present  the  results  in  a  format  suitable  for  decision  makers  and 
expert  systems  designers. 

3.2  RESEARCH  OBJECTIVES 

There  are  three  objectives  in  this  research  project.  The  first  is 
to  demonstrate  the  validity  the  Guilford  Structure  of  Intellect  model 
for  mapping  characteristics  of  task  domains.  The  second  is  to  develop  a 
representation  of  the  existing  capabilities  of  expert  systems  in  the  SOI 
model.  The  third  is  to  develop  and  validate  mathematical  algorithms  for 
comparing  domains  against  one  another  and  against  expert  systems 
capabilities . 

3.3  SYSTEM  OUTPUT 

The  output  of  the  domain  adviser  tool  will  be  in  three  parts.  Each 
part  (level)  of  the  output  is  designed  to  provide  the  user  with 
information  about  the  domain  of  interest  and  how  well  an  expert  system 
could  be  developed  for  that  domain. 
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LEVEL  ONE  ANALYSIS 


The  level  one  analysis  should  provide  the  decision  makers  with 
information  as  to  the  viability  of  expert  system  application. 

LEVEL  TWO  PROBLEM  AREA  DESCRIPTION  /  CAPABILITY 
DIFFERENCE  ANALYSIS 

The  level  two  analysis  should  allow  the  system  designer  to 
understand  the  characteristics  of  the  domain  of  interest  and  how  these 
characteristics  relate  to  expert  systems  capabilities. 


LEVEL  THREE  RECOMMENDATIONS  AND  COMMENTS 

The  level  three  output  of  the  analysis  will  add  additional  details 
to  the  information  presented  in  the  first  two  levels  of  analysis. 
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4.0  METHOD  AND  PROCEDURE 


4.1  DEVELOPMENT  OF  EXPERT  SYSTEMS  REPRESENTATION 

One  of  the  most  important  aspects  of  this  research  effort  is  the 
development  of  the  expert  systems  capability  representation.  This 
representation  is  important  tor  two  reasons.  First,  the  expert  systems 
representation  is  key  to  the  excursion  analysis  part  of  the  domain 
analysis  tool.  The  second  reason  the  expert  systems  representation  is 
important  is  that  there  is  currently  no  closed  form  analytical 
representation  of  expert  systems  capabilities. 

4.1.1  APPROACH 

In  order  to  develop  the  expert  systems  representation,  a  two-step 
procedure  was  followed.  The  first  step  was  to  review  the  existing 
knowledge  of  expert  systems  and  their  capabilities  as  embodied  by  the 
literature  on  the  subject.  This  review  yielded  a  great  deal  of 
important  information  which  was  incorporated  into  the  expert  systems 
capability  representation.  There  were,  however,  a  large  number  of  areas 
which  were  not  covered  by  this  literature  survey.  The  second  step  in 
modeling  expert  systems  capabilities  was  to  fill  in  a  complete  model  of 
the  expert  system  in  terms  of  the  SOI  model.  To  accomplish  this, 
information  was  solicited  from  experts  in  the  development  of  expert 
systems.  To  this  end,  a  structured  survey  questionnaire  was  developed 
to  solicit  expert  systems  capabilities  in  the  SOI  model. ng  space. 
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4.1.2  EXISTING  EVALUATIONS 


For  a  number  of  years,  there  has  been  a  large  number  of  designers 
and  researchers  developing  expert  systems.  By  doing  a  well-directed 
literature  search,  we  hoped  to  gain  an  understanding  of  what  these 
people  were  able  to  learn  over  the  last  years. 

4. 1.2.1  EVALUATION  OF  EXPERT  SYSTEMS  (SPECIFIC  CAPABILITIES) 

Expert  systems  are  good  at  a  lot  of  things.  One  way  of  evaluating 
the  current  capabilities  of  expert  systems  is  to  evaluate  the  tasks/ 
domains  that  they  currently  do  well.  In  fact,  it  is  very  common  in 
practice  today  to  evaluate  a  domain  based  on  its  similarity  to  what 
expert  systems  do  well.  Therefore,  by  evaluating  the  kinds  of  tasks 
expert  systems  do  well  in  terms  of  their  components  in  the  SOI  modeling 
space,  we  can  learn  about  expert  systems'  capabilities. 

Expert  systems  : 

Plan 

Schedule 

Diagnostics 

Configuration 

Modeling  and  Simulation 

Monitoring  and  Control 

Within  these  categories  there  have  been  a  number  of  successful  and 
unsuccessful  expert  systems  development  projects.  The  results  of  the 
evaluations  are  summarized  in  terms  of  the  SOI  model.  Very  little  was 
learned  about  failure  of  expert  systems. 

Planning:  Devising  a  method  for  making  or  doing  something  in  order 

to  achieve  an  end.  Applications  include  construction  projects,  regular 
delivery  routes,  and  manufacturing  schedules  (Westinghouse)  (Wolfgram  et 
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al,  1987).  These  tasks  show  that  expert  systems  will  perform  the 
convergent  production  operators  and  products  of  relations  and  systems. 

Scheduling:  Designing  routes  and  time  tables  for  events. 

Applications  include  aircraft  scheduling,  scheduling  of  medical 
treatments  and  scheduling  equipment  maintenance  (Pham,  1985).  An 
example  of  this  type  of  system  is  OPAL,  an  expert  system  that  schedules 
cancer  patients  for  chemotherapy.  These  tasks  show  that  expert  systems 
will  perform  in  the  areas  of  convergent  production  and  memory. 

Diagnostics:  Identifying  causes,  given  symptoms.  Applications 

include  diagnosing  infectious  diseases,  telephone  networks,  and 
poisoning  (Luger,  Stubblefield,  1984).  Included  in  these  applications 
are  MYCIN  and  DENDRAL  which  are  medical  diagnosis  expert  systems.  These 
characteristics  indicate  that  expert  systems  should  perform  well  in  the 
areas  of  symbolic  and  semantic  contents  as  well  as  memory  and  evaluation 
operators . 

Configuration:  Configuring  objects  into  systems,  given 

constraints.  Applications  include  computer  system  configuration,  layout 
of  computer  microchips,  and  factory  floor  layouts  (Bielawsky,  Lewand, 
1984).  The  development  systems  include  XCON,  a  system  developed  by  DEC 
for  configuring  VAX  computer  systems.  This  would  indicate  that  expert 
systems  will  work  well  in  relations  and  systems  of  products. 

Modeling  and  Simulation:  Simulation  of  existing  physical  relations. 
Applications  include  problem  solving  CAD/CAM  systems  and  simulation  of 
power  plants  (Woods,  Hollnagel,  1988).  These  examples  show  that  expert 
systems  can  perform  tasks  requiring  memory  and  evaluation,  as  well  as, 
symbolic  and  semantic  contents. 
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Monitoring  and  Control:  Comparing  observations  with  established 
standards  and  regulating  operation.  Applications  include  monitoring 
power  plants,  autonomous  land  vehicle  navigation,  and  manufacturing 
plants  (Costea,  1979).  These  applications  imply  that  expert  systems 
will  perform  well  in  the  areas  of  convergent  production  and  memory. 

4. 1.2.2  EVALUATION  OF  EXPERT  SYSTEMS  (GENERAL  CAPABILITIES) 

Characteristics  of  expert  systems  domains  are  documented  in  the 
literature  as  rules  for  selecting  expert  systems  domains.  The  best  and 
most  often  used  of  these  rule  sets  is  that  written  by  David  Prerau 
(1989)  in  "Choosing  an  Expert  System  Domain." 

In  this  section  we  review  these  rules  as  they  relate  to  the  SOI 
model  of  expert  systems  capabilities.  It  is  important  to  note  that 
there  are  a  number  of  methods  similar  to  Prerau’ s  which  were  studied. 
Prerau 's  recommendations  are  presented  because  they  are  so  commonly  used 
in  design.  In  "Choosing  an  Expert  System,"  he  states,  "To  evaluate  the 
potential  of  a  possible  application,  it  has  proven  very  useful  to  have  a 
set  of  the  desired  attributes  of  a  good  expert  system  domain."  Prerau 
goes  on  to  list  a  number  of  these  attributes,  which  are  listed  below  and 
evaluated  as  to  their  implications  to  the  SOI  model  of  expert  systems 
capabilities . 

"The  task  primarily  requires  symbolic  reasoning." 

This  attribute  clearly  implies  that  expert  systems  are  capable  of 
performing  tasks  with  symbolic  contents. 

"The  task  requires  the  use  of  heuristics.” 

This  attribute  implies  expert  systems  are  capable  of  performing 
operations  of  convergent  production. 
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"The  task  does  not  require  Knowledge  from  a  very  large  number  of 
areas  and  it  does  not  involve  common  sense  reasoning." 

This  attribute  implies  that  certain  behavioral  contents  and  implications 
are  not  well  suited  to  expert  systems. 

"The  task  is  defined  as  clearly  as  possible." 

This  attribute  implies  that  expert  systems  will  generally  not  perform 
well  in  cases  of  divergent  production. 

"The  expert  has  built  up  expertise  over  a  long  period  of 
task  performance." 

This  attribute  implies  that  expert  systems  are  well  suited  to  complex 
memory  tasks. 

"The  task,  and  preferably  every  subtask  is  decomposable." 

This  attribute  implies  that  an  expert  system  will  work  well  with  tasks 
involving  systems  and  relationships. 

There  are  a  large  number  of  other  attributes  listed  by  Prerau. 

Many  of  the  rest,  however,  deal  with  issues  which  are  not  addressed  in 
this  work,  such  as  availability  of  domain  experts  and  level  of 
management  commitment  to  the  project. 


4.1.3  EVALUATIONS  OF  DOMAIN  EVALUATION  TOOLS 

Most  notable  among  evaluation  tools  is  "Methodology  for  Screening 
Potential  Artificial  Intelligence  Applications"  by  Eric  Hanson  and 
Stephen  Cross  (1988).  In  this  methodology,  they  evaluate  a  domain  of 
interest  based  on  three  overall  parts  of  the  design  and  implementation 
of  the  expert  system:  level  of  AI  risk,  systems  engineering  risk,  and 
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potential  payoff.  There  are  sub-categories  under  each  of  these  general 


parts.  The  domain  is  evaluated  based  on  how  well  it 
arbitrary  implementation  parameters.  The  subclasses 
are: 


-  AI  Risk 

Type  of  Application 

Nature  and  Availability  of  Expertise 
Complexity  and  Difficulty  of  Task 
Role  of  the  System 

Size  and  Complexity  of  Knowledge  Base 
Applicability  of  current  AI  Tools 
Advanced  Technology  Requirements 

-  Systems  Engineering  Risk 

System  Complexity 
System  Scalability 
Performance  Requirements 
Hardware  Requirements 
Software  Requirement 
Maintainability 


-  Value  and  Cost 

Economic  Value 
Effectiveness 
Generality 
Cost 


fits  these 
of  the  evaluation 
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"A  mark  is  made  on  each  major  scale  that  essentially  summarizes  the 
specific  items  under  it.  Judgment  is  required  to  summarize  the  marks  on 
the  detailed  scales  to  come  up  with  an  assessment  for  the  major 
categories."  (Hanson  and  Cross,  1988) 

It  is  important  to  remember  when  reviewing  evaluation  tools  that 
these  tools  address  a  number  of  areas  which  the  ESDAAT  does  not  address. 
The  ESDAAT  is  principally  a  tool  to  evaluate  a  domain  on  its  technical 
characteristics  as  they  relate  to  expert  systems  capabilities. 

Included  in  Appendix  E  are  two  other  domain  evaluation  tools. 

These  two  tools  are  derived  from  the  tool  which  was  just  reviewed. 

For  the  purpose  of  developing  the  expert  system  capability 
representation,  these  tools  are  of  little  or  no  use  except  as  an  example 
of  what  is  not  needed. 

Summary 

From  an  analysis  of  the  current  literature  and  existing 
evaluations,  it  is  clear  that  expert  systems  function  well  in  some  parts 
of  the  SOI  model  space. 

Contents:  Symbolic  and  Semantic 

Operators:  Memory  and  Convergent  Production 

Products:  Relations,  Systems,  and  Units 

The  analysis  also  shows  that  expert  systems  do  not  function  at  all 
well  in  some  other  parts  of  the  SOI  model  space. 

Contents:  Auditory  and  Behavioral 

Operators:  Divergent  Production,  Cognition,  and 
Evaluation 

Products:  Transformations  and  Implication 

The  information  collected  in  this  review  is  used  in  conjunction 
with  that  obtained  in  the  knowledge  engineer  survey  interviews  to  create 
the  SOI  model  of  current  expert  systems  capabilities  shown  in  Figure  7. 
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4.1.4  DERIVED  EVALUATION 

4. 1.4.1  DEVELOPMENT  OF  THE  KNOWLEDGE  ENGINEER  SURVEY 

The  purpose  of  the  knowledge  engineering  survey  was  to  elicit  from 
expert  system  designers  a  concise  description  of  expert  systems 
capabilities.  In  order  to  obtain  the  most  useful  description  from  the 
expert  systems  experts,  the  survey  was  constructed  to  determine  a  number 
of  separate  items. 

In  Section  I  of  the  survey,  the  administrator  explains  the  nature 
of  the  development  project  and  the  nature  of  the  SOI  model. 

In  Section  II  of  the  survey,  the  knowledge  engineers  explain  their 
experience  in  developing  systems.  The  selection  of  expert  knowledge 
engineers  was  done  in  such  a  way  as  to  maximize  the  usefulness  of  the 
analysis  by  including  knowledge  engineers  with  the  most  experience. 

In  Section  III  of  the  survey,  the  knowledge  engineers  evaluate  Borne 
of  the  expert  systems  they  have  worked  on  in  order  to  give  them  a  chance 
to  think  about  how  to  evaluate  the  nature  of  expert  systems  capabilities 
in  terms  of  the  SOI  model. 

In  Section  IV,  the  knowledge  engineers  are  asked  to  rank  in  order 
the  elements  of  the  SOI  model  with  respect  to  the  capabilities  of  expert 
systems.  In  this  section  there  are  also  questions  to  solicit  general 
comments  on  which  elements  are  easy  or  difficult  for  expert  systems  to 
perform. 

In  Section  V,  the  knowledge  engineers  are  asked  to  rate  the 
individual  elements  and  combinations  of  elements  of  the  SOI  model  as  to 
their  criticality  to  construction  of  expert  systems.  In  this  way,  the 
knowledge  engineers  relate  which  of  the  elements  are  most  important. 
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In  section  VI,  the  knowledge  engineers  are  asked  to  evaluate  the 


elements  and  combinations  of  elements  of  the  SOI  model  based  on  the 
difficulty  level  at  which  expert  systems  can  perform  these  elements  and 
combinations.  In  this  way  we  get  a  direct  indication  of  the 
capabilities  of  the  expert  systems  to  perform  in  these  areas. 

In  section  VII,  the  knowledge  engineers  are  asked  to  relate 
additional  information  with  regards  to  the  possible  outputs  of  the  tool 
which  they  feel  would  be  most  useful  to  system  design  in  the  decision¬ 
making  process. 

The  survey  is  broken  into  two  parts.  The  first  part  is  for  the 
interviewer  and  helps  him  ask  the  questions,  explain  the  concepts  of  the 
project,  and  direct  the  interview.  The  second  part  is  the  answer  sheet 
for  the  knowledge  engineer  being  interviewed.  The  complete  knowledge 
engineering  survey  can  be  found  in  Appendix  A. 

4. 1.4. 2  DEVELOPMENT  OF  CRITICAL  ELEMENT  SURVEY 

The  purpose  of  the  critical  element  survey  is  to  solicit  additional 
information  from  the  domain  experts  on  the  most  important  aspects  of  the 
domain.  This  survey  is  given  to  domain  experts  after  the  DSAT  survey 
has  been  done  and  the  level  two  analysis  has  been  completed.  The  level 
two  analysis  includes  thresholding  to  determine  the  most  important  parts 
of  the  domain  so  that  we  can  solicit  more  information  about  these  parts 
of  the  domain  from  the  domain  experts.  The  results  of  the  critical 
element  survey  constitute  the  level  three  output  from  the  tool. 

The  critical  element  survey  is  divided  into  four  sections.  In  the 

first  section,  the  domain  expert  is  asked  to  evaluate  the  influence  of 

the  critical  element.  In  the  second  part  of  the  survey,  the  domain 

expert  is  asked  to  describe  the  critical  element  in  terms  of  its 
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criticality,  flexibility,  difficulty,  and  frequency.  The  third  part  of 
the  survey  asks  the  domain  expert  to  describe  the  nature  of  the 
expertise  in  this  area.  These  questions  are  used  to  define  the 
requirements  for  eliciting  knowledge  about  these  critical  elements 
within  the  domain  of  interest.  The  last  part  of  the  survey  asks  the 
domain  expert  to  evaluate  the  usefulness  of  an  expert  system  developed 
without  the  critical  element  being  examined. 

The  output  from  the  critical  element  survey  will  give  systems 
designers  important  specific  information  about  those  parts  of  the  domain 
which  may  cause  difficulty  in  the  development  of  a  useful  expert  system. 
The  complete  critical  element  survey  is  listed  in  Appendix  B. 
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4.2  DEVELOPMENT  OF  ANALYSIS  METHODOLOGY 


The  primary  considerations  taken  into  account  in  the  development  of 
the  analysis  tool  were  the  validity  of  the  analysis  and  the  operational 
considerations.  The  development  of  the  analysis  tool  was  also  based  on 
the  project  goals,  as  stated  in  Section  3,  and  on  the  desired  system 
outputs.  The  level  one  output  is  developed  using  the  neural  network. 

The  level  two  output  is  based  on  excursion  analysis  and  parametrizes  the 
differences  between  expert  systems  capabilities  and  the  requirements  of 
a  given  domain.  The  level  three  output  focuses  on  providing  additional 
information  about  the  most  critical  areas  of  differences  between  the 
capabilities  of  expert  systems  and  the  requirements  of  the  domain. 

4.2.1  DEVELOPMENT  OF  THE  LEVEL  ONE  ANALYSIS 

Neural  Net 

There  were  two  considerations  in  determining  the  form  of  the  output 
level  one  analysis.  The  first  was  to  provide  information  that  would  be 
of  the  most  use  to  the  system  designers  and  project  managers.  The 
second  consideration  was  to  present  the  most  information  about  a  domain 
as  is  practical  in  a  condensed  form. 

Based  on  these  criteria  and  the  recommendations  of  the  expert 
knowledge  engineers  who  were  interviewed,  the  level  one  analysis 
presents:  Probability  of  Success,  Difficulty,  and  Resources  Required. 

The  dimensions  of  the  level  one  analysis  are  meant  to  be 
orthogonal  and  describe  different  parts  of  the  design  and  implementation 
of  an  expert  system.  Probability  of  success  is  meant  to  represent  the 
basic  compatibility  of  the  domain  with  the  technology  of  expert  systems. 
Difficulty  is  meant  to  express  the  complexity  of  the  domain  and  the 
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amount  of  difficulty  a  developer  will  have  in  implementing  an  expert 
system  to  perform  satisfactorily  in  this  domain.  Resources  required  is 
a  function  of  number  and  experience  level  requirements  of  the  domain 
experts  and  systems  designer  who  will  implement  the  expert  system. 
Therefore,  this  is  a  multi-dimensional  classification  problem.  While 
traditional  statistical  methods  might  be  used  for  this  classification 
problem,  the  multi-dimensional  character  of  the  problem  makes  a  neural 
net  classifier  more  desirable. 

During  the  initial  development  of  the  level  one  analysis, 
consideration  was  given  to  the  development  of  a  simple  decision  rule  to 
determine  if  an  expert  system  is  appropriate.  It  is  our  opinion,  and 
that  of  most  expert  system  designers  we  interviewed,  that  a  go-no  go 
glooal  decision  rule  would  be  of  very  little  use  to  designers.  Also, 
such  a  "global  decision  rule"  might  give  the  wrong  impression  to 
managers  that  the  decision  to  develop  an  expert  system  was  a  less 
complex,  multi-dimensional  question  than  we  know  it  to  be. 

4. 2. 1.1  NET  ARCHITECTURE 

In  developing  the  neural  net  we  looked  at  several  different 
architectures  and  learning  methods.  When  selecting  the  architecture  of 
the  net,  we  looked  at  three  different  input  training  sets:  1)  Inputting 
all  450  DSAT  raw  values,  2)  Inputting  a  reduced  set  of  DSAT  values,  and 
3)  Inputting  a  set  of  summary  statistics  about  the  domains. 

We  started  the  neural  net  development  with  the  plan  to  use  the  full 
450  vector  input  format  of  the  DSAT  survey.  This  net  proved  to  be 
unstable  and  very  difficult  to  train  (see  Figure  5). 
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The  second  possible  architecture  we  studied  is  based  on  a  reduced  set 


of  the  DSAT  parameters.  This  is  the  architecture  which  was  selected 
because  it  retained  the  diverse  character  of  the  DSAT  and  was 
implementable  and  stable  (See  Figure  6).  The  latter  architecture  we 
studied  was  a  net  based  on  summary  statistics  of  the  DSAT  survey.  This 
architecture  is  appealing  because  of  its  simplicity  but  it  removes  too 
much  of  the  resolution  of  the  domain  description. 

The  summary  statistics  are  a  first-level  data  reduction.  To  do 
the  data  reduction,  we  go  back  to  the  organization  of  the  data 
collection  device  (DSAT  questionnaire).  Based  on  the  first  part  of  the 
DSAT,  we  calculate  two  summary  numbers  for  each  of  the  content-product 
aspects  of  the  domain.  The  first  summary  number  is  the  number  of  active 
elements  in  the  block,  and  the  second  is  the  magnitude  of  the  elements 
in  the  block.  These  two  numbers  f  r  each  content-product  aspect  of  the 
domain  constitute  the  input  vector  for  the  second  net  architecture. 

A  three-layer  perceptron  was  chosen  to  implement  the  second 
architecture.  A  three-layer  perceptron  such  as  we  are  using  here  can 
form  arbitrarily  complex  decision  regions  and  can  separate  meshed 
classes  (Lippmann,  1987).  The  number  of  input  nodes  to  the  net  is  50, 
two  for  each  combination  of  operators  and  contents  in  the  domain 
description.  There  are  three  outputs  from  the  net  -  Probability  of 
Success,  Difficulty,  and  Resources  Required.  In  selecting  the  number  of 
nodes  in  the  second  and  first  layers,  we  again  refer  to  Lippmann.  "The 
number  of  nodes  in  the  second  layer  must  be  greater  than  one  when 
decision  regions  are  disconnected  or  meshed  and  cannot  be  formed  from 
one  convex  area.  The  number  of  second  layer  nodes  required  in  the  worst 
case  is  equal  to  the  number  of  disconnected  regions  in  input 
distributions.  The  number  of  nodes  in  the  first  layer  must  typically  be 
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sufficient  to  provide  three  or  more  edges  for  each  convex  area  generated 


by  every  second-layer  node.”  In  the  first  hidden  layer  there  are  six 
nodes  and  in  the  second  hidden  layer  there  are  also  six  nodes. 

4.2. 1.2  NET  TRAINING 

The  decision  on  a  training  algorithm  was  fairly  easy.  This  kind  of 
evaluation/classification  problem  is  very  well  suited  to  the  use  of 
back-propagation  training  (McClelland  and  Rumelhart,  1988).  The  back- 
propagation  algorithm  has  been  tested  with  a  number  of  deterministic 
problems.  It  has  been  found  to  perform  well  in  most  cases  and  to  find 
good  solutions  to  the  problems  posed.  The  training  values  which  were 
used  in  training  the  net  outputs  (probability  of  success,  difficulty, 
and  resources  required)  were  obtained  from  expert  knowledge  engineers 
who  estimated  the  output  parameters  for  the  training  domains. 

4.2.2  EXCURSION  ANALYSIS 

The  critical  element  (CE)  analysis  of  excursions  is  similar  to 

critical  incident  (Cl)  methods  commonly  used  in  behavioral  analysis.  Cl 

can  be  most  particularly  useful  as  the  basis  for  inferring  the  qualities 

or  attributes  relevant  to  successful  performance.  Therefore,  we  are 

deriving  a  similar  method  which  will  be  suited  to  analysis  of  how  expert 

systems  interact  with  the  domains  for  which  they  are  designed. 

One  of  the  potential  problems  with  using  a  Cl  or  CE  analysis  is 

that  there  is  very  little  information  about  the  validity  and  reliability 

of  the  analysis.  Many  of  these  concerns  are  resolved  by  using  the 

Structure  of  Intellect  model  and  the  DSAT  data  collection  tool.  By 

using  this  modeling  tool  to  quantify  the  decision  criteria,  the 

* 

ex  ursion  analysis  can  be  performed  on  a  one-to-one  basis  within  the  SOI 
modeling  space. 
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The  excursion  analysis  is  made  possible  by  the  development  of  the 


expert  systems  capability  representation.  The  expert  systems  capabilicy 
representation  defines  the  level  cf  difficulty  or  complexity  with  which 
an  expert  system  can  work  within  each  elemer.t  of  the  SOI  modeling  space. 
The  scale  of  expert  systems  capability  is  in  units  of  difficulty  ranging 
from  zero  to  ten.  Once  the  expert  systems  capabilities  and  tie  domain 
requirements  are  represented  on  the  same  scale  and  in  the  same  modeling 
space,  the  excursion  analysis  can  be  dote.  The  excursion  analysis 
consists  of  point-by-point  subtraction  of  expert  systems  capabilities 
from  domain  requirements.  The  remainder  is  then  the  level  of  domain 
requirements  which  are  not  covered  by  expert  systems  capabilities. 

In  evaluating  excursions,  it  is  useful  to  define  critical  elements 
of  the  excursion  analyses.  Critical  elements  can  be  defined  in  two 
possible  ways:  1 )  as  a  point  of  most  excursion  from  expert  systems 
capabilities  ,  or  2)  as  a  function  of  the  criticality  of  the  element  to 
the  successful  completion  of  the  domain  task. 

The  critical  elements  are  first  defined  as  all  of  the  excursion 
points.  The  excursion  points  are  then  examined  based  on  the  amount  or 
level  of  excursion  difference  between  capabilities  and  requirements. 

Then  the  criticality  of  the  elements  can  be  examined.  The  criticality 
of  each  element  is  assigned  based  on  the  data  collected  from  the  DSAT 
domain  description.  The  most  important  property  of  a  critical  element 
is  the  level  of  excursion  followed  by  the  criticality  of  the  element. 
Therefore,  the  critical  element  thresholding  algorithm  first  sorts  the 
excursion  points  by  level  of  excursion  and  then  by  criticality  of 
element . 

% 

Summary  statistics  are  also  calculated  for  two  reasons.  The  first 
reason  is  to  provide  a  quick  non-visual  means  of  comparing  domain 
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evaluations.  The  second  reason  is  to  provide  a  measure  of  how  domains 


relate  to  one  another  which  is  independent  of  the  level  one  analysis. 
This  will  aid  in  the  validation  of  the  methodology.  Two  summary 
statistics  are  calculated  for  the  level  two  analysis.  The  first  is  a 
count  of  the  number  of  excursions  a  domain  makes  from  the  expert  systems 
capabilities.  In  other  words,  this  is  a  count  of  the  number  of  elements 
in  which  the  domain  requirements  are  greater  than  the  expert  systems 
capabilities.  The  second  statistic  is  a  measure  of  the  amount  of 
excursion  and  criticality  of  the  excursion  points.  This  statistic  is 
called  the  critical  volume.  It  is  calculated  by  taking  the  sum  of  the 
difficulty  difference  multiplied  by  three  and  the  criticality  multiplied 
by  two  for  each  excursion  point. 

By  identifying  and  rank-ordering  the  criticality  of  domain  require¬ 
ments  which  exceed  expert  systems  capabilities  (excursion  points),  an 
additional  opportunity  exists  to  gather  valuable  information  about  the 
domain  based  on  the  most  critical  elements.  To  this  end,  the  critical 
element  list  is  used  to  select  elements  to  study  in  greater  depth. 

4. 2. 2.1  PLOTS  OF  DOMAIN  REQUIREMENTS 

The  output  of  the  level  two  analysis  consists  of  two  parts.  The 
first  part  of  the  output  is  the  chart  of  the  domain  requirements  which 
' 3  derived  directly  from  the  results  of  the  DSAT  description  of  the 
domain.  The  results  of  the  DSAT  questionnaire  are  formatted  into  the 
"raw  data".  The  difficulty  rating  for  each  element  is  extracted  from 
the  raw  data  and  plotted  on  the  domain  requirement  chart.  In  this  way 
the  user  of  the  tool  can  see  a  complete,  concise  representation  of  the 
domain  of  interest.  This  chart  is  very  useful  in  understanding  the 
general  nature  of  the  domain  and  making  certain  that  everyone  agrees  on 
the  nature  of  the  domain. 
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4. 2. 2. 2  PLOTS  OF  EXCURSIONS 


The  remainder  of  the  level  two  output  is  devoted  to  representing 
the  excursions.  The  excursions  are  first  presented  in  the  form  of 
another  capability  chart.  This  chart  will  represent  the  difference 
between  the  level  of  difficulty  which  the  domain  requires  and  the  level 
of  difficulty  that  an  expert  system  can  support.  This  difference  chart 
presents  the  excursion  points  in  a  concise  manner.  From  this  chart  the 
user  can  see  the  parts  of  the  domain  which  cannot  be  supported  by  expert 
systems  technology.  The  last  part  of  the  level  two  output  is  the 

listing  of  critical  elements.  In  this  listing,  the  excursion  points  are 

listed  in  order  of  their  criticality  to  the  domain  and  the  expert  system 
implementation. 

4.2.3  CRITICAL  ELEMENT 

The  output  of  the  critical  element  analysis  will  be  the  level  three 
output  to  the  user.  The  critical  element  analysis  is  a  questionnaire 
based  on  obtaining  additional  information  from  the  domain  experts  on  the 
part  of  the  domain  with  which  we  can  reasonably  expect  the  expert  system 
development  project  to  have  difficulty.  The  questions  on  the  critical 
element  survey  were  designed  to  address  the  key  issues  that  will  effect 
the  development  and  effectiveness  of  the  expert  system. 

The  choice  of  which  elements  to  input  into  the  critical  element 

survey  process  is  determined  from  the  level  two  excursion  analysis.  The 
elements  which  differ  most  from  the  expert  system  capabilities  are 
listed,  starting  with  elements  with  difficulty  differences  of  ten  and 
criticality  of  ten  and  continuing  through  the  excursion  list  as  long  as 
there  are  resources  to  perform  the  survey. 
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There  is  no  additional  analysis  done  of  the  critical  element  survey 
output  data. 

Results  of  the  Prototyping 

In  order  to  determine  the  usefulness  of  the  critical  element 
survey,  the  survey  was  given  to  a  domain  expert  in  the  domain  of  Mission 
Planning.  The  results  of  the  survey  for  the  two  most  important  critical 
elements  are  listed  in  Appendix  B.l  and  B.2.  As  can  be  seen  from 
examining  the  results  of  the  critical  element  survey  prototyping,  a 
great  deal  of  important  information  about  these  critical  elements  can 
learned  from  the  critical  element  survey. 
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Figure  5 
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UCED  INPUT  VECTOR 


4 . 3  PROCEDURE 


The  procedures  followed  in  the  data  collection  of  this  research  are 
presented  in  this  section.  The  data  acquisition  devices  include:  1)  the 
DSAT  survey  developed  by  Jay  Horn  and  2)  the  knowledge  engineer  and 
critical  element  surveys  developed  is  in  Section  4.  The  analysis 
procedures  are  also  described. 

4.3.1  KNOWLEDGE  ENGINEERS  SURVEY 

In  order  to  obtain  this  representation,  it  was  determined  that  five 
expert  knowledge  engineers  would  be  interviewed  and  their  responses 
summarized  to  create  the  capability  representation.  The  expert 
knowledge  engineers  were  selected  based  on  their  level  of  experience, 
diversity  of  experience  and  availability.  The  expert  knowledge 
engineers  who  were  selected  had  experience  in  the  areas  of  medicine, 
maintenance,  software  engineering,  systems  evaluation,  and  AI  research. 
Each  of  the  knowledge  engineers  chosen  has  extensive  experience  in  AI, 
computer  systems  design,  and  expert  system  development.  Their  average 
amount  of  experience  in  expert  systems  is  eight  years  and  four  of  the 
five  have  doctorate  degrees.  The  survey  required  about  two  hours  to 
complete. 

4.3.2  DOMAIN  SURVEY 

The  domain  surveys  were  administered  to  domain  experts  following 
the  procedures  listed  in  Jay  Horn's  thesis.  The  DSAT  survey  was  used  as 
the  data  acquisition  device  for  the  domain  surveys.  There  were  a  total 
of  eight  domains  for  which  the  DSAT  data  was  collected.  These  domains 
and  the  criteria  for  their  selection  are  listed  in  Section  5.  The  DSAT 
questionnaire  required  approximately  two  to  three  hours  for  each  of  the 
domain  experts  to  complete.  The  domain  experts  were  very  helpful  and 
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cooperative.  The  domain  experts  did,  however,  require  a  good  deal  of 
explanation  of  the  SOI  model  in  order  to  complete  the  survey. 

4.3.3  ANALYSIS 

The  analysis  of  the  results  of  the  knowledge  engineers  survey 
consists  of  summarizing  the  rank  ordering  and  capability  evaluations  in 
the  survey.  The  results  of  this  analysis  are  presented  in  Section  8. 

The  data  analysis  of  the  domain  survey  data  consists  of  presenting  the 
data  to  the  neural  net  in  order  to  obtain  the  global  recommendations  and 
to  run  the  level  two  analysis.  The  level  two  analysis  or  excursion 
analysis,  described  in  Section  4.2,  consists  of  running  the  DSAT  raw 
data  through  a  number  of  FORTRAN  and  SAS  computer  programs.  The  results 
of  the  analysis  of  the  domain  data  are  presented  in  Section  5. 

4.3.4  CRITICAL  ELEMENTS 

After  the  analysis  was  completed  on  all  of  the  data,  a  domain  was 
selected  to  use  for  prototyping  of  the  critical  element  survey.  The 
domain  of  mission  planning  was  selected  because  it  is  of  medium 
difficulty  and  because  the  thresholding  showed  several  interesting 
elements  to  prototype.  After  the  mission  planning  domain  was  selected, 
the  critical  element  survey  was  administered  to  a  domain  expert  for  the 
top  two  critical  elements.  The  results  of  the  critical  element  surveys 
are  discussed  in  Section  4.2.3  and  presented  in  Appendices  B.l  and  B.2. 
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5.0  RESULTS 


5.1  DATA  COLLECTION 

The  data  collection  was  conducted  in  a  manner  consistent  with 
Section  4.3.  The  subjects  in  the  data  collection  were  extremely 
enthusiastic  and  contributed  significantly  to  this  research. 

5.1.1  KNOWLEDGE  ENGINEERS  SURVEY 

The  knowledge  engineers  survey  was  conducted  in  order  to  obtain  the 
necessary  information  to  complete  the  description  of  expert  systems 
capabilities  within  the  SOI  modeling  domain.  Sections  IV,  V,  and  VI  of 
the  questionnaire  address  this  description.  The  results  of  the 
knowledge  engineers  survey  is  summarized  in  Appendix  A.l.  The  results 
of  Section  IV,  which  ranks  the  parameters  of  three  dimensions  of  the  SOI 
model,  are  combined  with  the  results  of  Sections  V  and  VI,  which  give 
numerical  ratings  for  the  individual  SOI  elements,  to  create  the 
capability  chart  for  expert  systems  (see  Figure  7).  In  Figure  7,  there 
is  a  histogram  of  the  capabilities  of  expert  systems  to  perform  task 
elements  at  certain  levels  of  difficulty  of  the  task  element.  The  scale 
runs  from  0  to  10  for  each  element  consisting  of  an  operator,  content, 
and  product.  A  scale  reading  of  10  means  that  expert  systems  are 
capable  of  performing  at  varying  levels  of  difficulty  and  complexity  to 
meet  demands  in  this  element  of  the  model.  A  scale  reading  of  0  means 
there  is  little  or  no  capability  within  expert  systems  to  perform  within 
the  model  element.  Other  sections  of  the  survey  results  are  addressed 
in  Sections  4  and  6. 
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Additional  Results 


One  of  the  other  important  problems  in  developing  expert  systems  is 
the  selection  of  the  software  design.  There  are  a  number  of  different 
approaches  to  designing  a  rule-based  system,  including  forward  chaining, 
backward  chaining,  case-based  reasoning,  etc.  The  selection  of  design 
is  based  on  the  characteristics  of  the  domain.  It  was  noted  by  several 
of  the  knowledge  engineering  experts  that  the  information  obtained  in 
the  domain  evaluation  could  be  constructively  used  for  recommending  a 
design  approach  for  the  expert  system  implementation  (See  Section  6.2). 

5.1.2  COMPARISON  WITH  LITERATURE  REVIEW 

The  review  of  expert  systems  literature  and  recommendations 
provided  a  great  deal  of  information  as  to  the  nature  of  the 
capabilities  of  expert  systems  (See  Section  4.1).  This  information, 
however,  was  incomplete  with  regards  to  modeling  expert  systems 
capabilities  in  the  SOI  modeling  space.  Therefore,  the  information  from 
the  literature  review  serves  to  reinforce  the  model  which  was  developed 
based  on  the  knowledge  engineers  survey  (See  Figure  7). 

5.1.3  DOMAIN  DATA  COLLECTION 

The  domain  data  collection  used  the  DSAT  questionnaire  to  obtain 
descriptions  of  various  domains  of  interest.  The  data  collection  was 
conducted  as  described  in  Section  4.3.  Domain  data  was  collected  for 
six  development  domains;  computer  program  debugging,  gifted  child 
assessment,  writing  research  proposals,  mission  planning,  electrical 
design,  and  medical  diagnosis.  Domain  data  was  also  collected  for  two 
test  domains,  musical  performance  and  troubleshooting  electrical 
equipment.  The  reasons  and  decision  criteria  for  selecting  these 
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domains  is  described  in  Section  5.4.  The  domain  characteristics  are 
represented  in  domain  requirement  plots,  which  are  contained  in  Appendix 
C.l  and  make  up  the  first  section  of  the  level  two  analysis. 

5.2  NEURAL  NET  (LEVEL  ONE  ANALYSIS ) 

The  neural  nets  were  constructed  and  trained  in  a  manner  described 
in  Section  6.1,  using  net  modeling  algorithms  described  by  McClelland 
and  Rumelhart  (1988). 

5.2.1  NET  PERFORMANCE  IN  TRAINING  AND  CONVERGENCE 

The  first  net  we  considered  was  Net  A  with  a  complete  input  vector 
which  included  all  450  input  values  from  the  DSAT  raw  data.  The 
training  performance  of  the  net  is  plotted  in  Figure  8.  This  net  proved 
to  be  unstable  in  training.  There  was  also  advice  from  several  neural 
net  experts  during  this  part  of  the  research  to  the  effect  that  Net  A 
had  an  inferior  design.  Net  B,  which  used  a  reduced  input  vector, 
proved  to  be  a  very  reasonable  design  and  was  chosen  as  the  final  net 
for  the  level  one  analysis.  The  training  performance  for  Net  B  is 
plotted  in  Figure  9. 

The  final  net  architecture  is  a  50  input,  3  output  back-propagation 
net  with  two  layers  of  hidden  nodes  (see  Figure  6).  The  net  converged 
after  2,138  iterations  with  a  final  mean-squared  error  of  0.023%  (See 
Figure  9).  The  final  output  values  for  the  level  one  analysis  are 
listed  in  Table  3.  The  final  inter-connection  weights  for  the  net  are 
listed  in  Appendix  C.l. 

5.2.2  DISCUSSION 

After  the  net/level  one  analysis  was  completed,  the  input  data  was 
run  in  the  level  one  analysis.  The  results  of  the  analysis  are  listed 
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in  Table  3  and  plotted  in  Figure  13.  The  results  clearly  show  that  some 
domains  are  much  better  suited  to  an  expert  system  implementation  than 
others.  In  order  to  test  the  sensitivity  and  usefulness  of  the  level 
one  analysis,  two  test  domains  were  also  run  through  the  level  one 
analysis.  The  results  are  plotted  along  side  the  other  domains  in 
Figure  14. 

5.2.3  INTERPRETING  RESULTS 

The  meaning  of  the  scales  for  the  level  one  analysis  are  based  on 
useful  interpretation  of  the  scale  parameters  as  they  relate  to  the 
design  and  implementation  of  expert  systems. 

Probability  of  Success 

A  score  of  0  in  probability  of  success  indicates  that  building  an 
expert  system  for  this  domain  would  be  practically  impossible.  A  score 
of  100  on  probability  of  success  indicates  a  domain  which  is  ideally 
suited  to  an  expert  system  solution. 

Difficulty 

A  score  of  0  on  difficulty  indicates  that  the  design  of  an  expert 
system  in  this  domain  would  be  very  easy  and  require  very  little 
expertise  in  expert  systems.  A  score  of  100  on  difficulty  indicates 
that  the  difficulty  of  the  domain  will  make  it  almost  impossible  to 
develop  an  expert  system. 

Resources  Required 

A  score  of  0  on  resources  required  indicates  that  the  resources 
required  will  be  minimal;  only  an  novice  in  the  field  will  be  required 
and  only  the  simplest  of  designs  will  be  required  (a  toy  problem).  A 
score  of  100  on  resources  required  indicates  that  resources  required  to 
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implement  an  expert  system  in  this  domain  tends  towards  infinity.  The 
practicality  of  designing  and  implementing  an  expert  system  in  this 
domain  is  eliminated  by  the  overwhelming  resources  required  to  do  the 
design  task  (in  terms  of  domain  experts  and  design  complexity). 

General izability 

It  has  been  shown  that  a  multi-layer  perceptron  trained  by  a  back- 
propagation  algorithm  is  mathematically  equivalent  to  an  optimum 
Bayesian  estimator.  It  has  also  been  shown  that  the  neural  network  will 
perform  as  an  optimum  estimator  on  any  arbitrary  general  input  vector 
provided  that  the  training  set  constitutes  a  spanning  set  of  the  region 
in  which  the  classification  takes  place.  Therefore,  if  the  training  set 
of  domains  can  be  said  to  span  the  region  of  the  domains  of  interest, 
then  the  network  will  be  guaranteed  to  give  an  optimum  estimation  of  the 
mapping. 

5.3  EXCURSION  ANALYSIS 

5.3.1  DATA 

A  level  two  data  analysis  was  performed  on  the  raw  data  collected 
for  all  of  the  domains.  This  data  and  the  expert  systems  capability 
representation  make  the  excursion  analysis  possible.  The  input  data  for 
each  domain  can  be  shown  on  a  domain  requirements  chart  such  as  the  one 
for  the  domain  of  writing  research  proposals  which  is  shown  in  Figure 
10.  The  remainder  of  the  capability  charts  are  listed  in  Appendix  C.l. 

Capability  requirements  charts  such  as  the  one  shown  in  Figure  10 
are  laid  out  in  the  same  manner  as  the  expert  system  requirement  chart 
shown  in  Figure  7.  In  the  capability  requirements  charts,  however,  the 
entries  represent  the  level  of  difficulty  that  a  given  domain  requires 
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for  a  complex,  sufficient  solution.  By  designing  these  representations 


in  this  way,  we  can  compare  numerically  the  requirements  of  a  domain 
against  the  capabilities  of  expert  systems  to  meet  these  requirements. 

5.3.2  PLOTS  AND  CRITICAL  ELEMENTS 

The  level  two  output  for  the  domains  for  which  we  collected  data 
includes  the  domain  capability  charts,  the  difference  charts,  the 
listings  of  the  critical  elements,  and  the  level  two  summary  statistics. 
The  summary  statistics  for  the  level  two  analysis  are  listed  in  Table  3. 
An  example  of  the  difference  chart  and  the  critical  element  listing  are 
shown  in  Figures  11  and  12.  The  remainder  of  these  charts  and  tables 
are  listed  in  the  appendix. 

The  difference  charts,  like  the  one  shown  in  Figure  11,  represent 
the  difficulty  difference  for  each  element  of  the  SOI  model  between  the 
requirement  of  the  domain  and  the  capability  of  expert  systems.  What  is 
plotted,  therefore,  is  the  level  of  difficulty  which  the  domain  requires 
above  and  beyond  what  an  expert  system  can  perform. 

The  critical  element  listing  is  a  listing  of  all  the  elements  of 
the  model  which  appear  on  the  difference  chart.  The  elements  where 
domain  requirements  are  greater  than  expert  system  capabilities  are 
sorted  by  difficulty  difference  and  then  by  criticality  of  the  element. 
In  this  way,  the  system  evaluator  can  review  the  most  critical  elements 
which  exceed  the  capability  of  expert  systems. 

5.3.3  DISCUSSION 

The  level  two  analysis  was  designed  with  the  expert  system  designer 
in  mind  as  opposed  to  the  level  one  analysis  which  addresses  managers. 

In  the  difference  charts,  there  is  a  great  deal  of  important  information 


57 


which  the  system  designer  can  use  in  developing  an  approach  to 
implementing  or  abandoning  an  expert  system  project.  An  example  of  this 
is  the  research  proposal  domain.  By  looking  at  the  difference  chart,  we 
can  see  that  the  domain  requires  more  in  the  area  of  semantic  contents 
than  expert  systems  are  typically  capable  of.  A  design  would  have  to 
find  a  method  of  either  altering  the  domain  characteristics  or  making  a 
breakthrough  in  expert  systems  design  to  develop  this  solution.  As  can 
be  seen  from  the  difference  charts  and  critical  element  listings,  there 
are  considerable  differences  between  the  different  domains.  The 
population  of  difference  charts,  however,  gives  a  clear  indication  of 
how  well  an  expert  system  will  suit  the  domain. 

5.4  VALIDATION  OF  ESDAAT 

5.4.1  DOMAIN  SELECTION 

The  original  three  domains  for  which  Jay  Horn  collected  raw  data 
provide  the  base  cases  for  the  validation  of  the  level  one  analysis  and 
the  excursion  analysis.  The  analysis  of  these  domains  is  shown  along 
with  the  other  domains  in  Section  5.2. 

After  the  expert  systems  capability  model  was  developed,  it  was 
determined  that  the  original  three  domains  (computer  program  debugging, 
writing  research  proposals,  and  gifted  child  assessment)  do  not 
sufficiently  cover  all  the  elements  of  the  Structure  of  Intellect  model 
and  expert  systems  capabilities.  The  original  three  domains  do  not  have 
any  coverage  at  all  of  visual  divergent  production  and  behavioral 
convergent  production  and  have  only  very  sparse  coverage  of  visual  and 
auditory  memory,  behavioral  evaluation,  and  divergent  production.  This 
lack  of  coverage  of  the  model  space  limits  the  theoretical 
generalizability  of  the  analysis.  The  lack  of  coverage  of  the  model 
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space  shows  that  there  are  additional  classes  of  domains  not  represented 


by  the  original  three  domains.  Therefore,  the  face  validity  of  the 
analysis  is  also  reduced. 

To  meet  the  need  for  a  greater  coverage  of  the  Structure  of 
Intellect  model  space,  we  chose  to  gather  DSAT  data  on  three  additional 
domains.  The  selection  of  additional  domains  was  made  on  the  basis  of 
three  criteria. 

(1)  That  the  additional  domains  complete  the  coverage  of  the  modeling 
space. 

(2)  That  the  additional  domains  be  significantly  different  from  the 
original  domains  in  order  to  increase  the  coverage  of  domains  in  general 

(3)  That  the  additional  domains  be  of  significant  interest  to 
potential  expert  systems  application. 

Based  on  these  criteria,  we  selected  the  following  domains. 

a.  Medical  Diagnosis 

b.  Mission  Planning 

c.  Electrical  Design 

The  addition  of  these  domains  has  significantly  increased  the 
portion  of  the  Structure  of  Intellect  model  which  is  covered.  With 
these  six  development  domains,  we  have  coverage  of  all  of  the  SOI 
modeling  space.  Complete  coverage  of  the  model  space  assures  a  level  of 
generalizability  of  the  analysis  tool.  Also,  by  including  six  widely 
diverse  domains  in  the  validation  of  the  analysis  tools,  we  greatly 
increase  the  face  validity  of  the  tool  in  general. 

For  the  capability  analysis  of  expert  systems  as  shown  in  Section 
4.1,  all  of  the  aspects  'e  expert  systems  capabilities  are  covered  or 
exceeded  by  one  or  more  of  the  development  domains. 
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Two  test  domain  were  selected  for  two  reasons,  first  to  provide  an 


unambiguous  test  of  the  level  one  analysis  and,  second,  to  provide  a 
wide  range  of  axis  anchors.  Therefore,  the  test  domains  should  be 
domains  where  expert  systems  clearly  are  or  clearly  are  not  good  choices 
for  a  solution  in  the  domain.  We  also  wish  the  test  cases  to  be  real 
domains  in  which  significant  expert  systems  development  has  taken  place 
or  been  attempted.  Based  on  these  criteria,  we  selected  the  domains  of 
electrical  equipment  troubleshooting  and  musical  performance.  The 
results  of  the  level  one  analysis  are  indicated  by  arrows  on  the  plot  of 
the  development/verification  domains  (See  Figure  14).  As  can  be  seen  in 
the  figure,  the  test  cases  form  a  very  useful  anchor  for  comparison  of 
other  domains  on  the  scale  of  probability  of  success.  On  the  scales  of 
difficulty  and  resources  required,  the  test  cases  are  considerably  less 
useful  as  scale  anchors.  This  is  to  be  expected,  as  the  test  domains 
are  less  complex  than  the  more  interesting  development  domains. 

5.4.2  EXCURSION  ANALYSIS  AND  NET  VERIFICATION 

When  evaluating  the  reliability  of  the  analysis  technique,  the 
level  one  analysis  outputs  and  the  level  two  analysis  summary 
statistics  can  all  be  compared  based  on  a  go/no  go  binary  decision  basis 
using  the  Wilcoxon  Two  Sample  Rank-Sum  test  by  comparing  the  orderings 
of  the  domains  (Lapin,  1973)  and  by  comparing  the  consistency  between 
the  analysis  methods. 

For  the  purpose  of  verification,  summary  parameters  were  calculated 
on  the  level  two  analysis.  These  parameters  are,  first,  the  number  of 
excursion  with  greater  than  3.3  difference  in  the  domain  requirements 
and  greater  than  level  four  criticality.  And  second,  a  critical  volume 
parameter,  calculated  by  three  times  the  difficulty  difference  plus  two 
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times  the  criticality.  These  parameters  are  listed  side-by-side  with 
the  output  of  the  level  one  analysis  in  Table  3. 

The  two  samples  are  grouped  by  scale  of  the  probability  of  success 
parameter  of  the  net  output. 

Group  1  >  50%  probability  of  success 
Group  2  <  50%  probability  of  success 

As  can  be  seen  from  the  Rank-Sum  test  (Table  4),  there  is 
considerable  agreement  between  all  of  the  analysis  methods. 
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TABLE  3 


l: vel  one  analysis 


TABLE  4 


WILCOXCN  TWO-SAMPLE 
RANK-SUM  TEST 


DOMAINS 

PROBILITY  OF 
SUCCESS 

RANK 

Troubleshooting 

Electrical 

Equipment 

95 

1 

Electrical  Design 

85 

O 

Computer  Program 
Debugging 

78 

3 

Mission  Planning 

60 

4 

Gifted  Chi  Id 

Assessment 

40 

5 

writing  Researcn 
Proposals 

20 

6 

Medical  Diagnoses 

1  1 

7 

Musical  Performance 

8 

8 

W  I  =  1  *2*3*4  =  I  0 
W2  =  5  *6*7*8  =  26 
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T ABLE  4 


WILCOXON  TWO-SAMPLE 
RANK-SUM  TEST 


DOMAINS 

RESOURCES 

REQUIRED 

RANK 

Troubleshooting 

Electrical 

20 

1 

Equipment 

Electrical  Design 

55 

3 

Computer  Program 

38 

Debugging 

) 

Mission  Planning 

71 

5 

Gifted  Chi  Id 

Assessment 

80 

6 

Writing  Research 
Proposals 

62 

4 

Medical  Diagnoses 

99 

8 

Musical  Performance 

76 

7 

W  1  =  I  +3+2+5  =  1  I 
W2  =  6+A+8+7  =  25 
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TABLE  4 


WILCOXON  TWO-SAMPLE 
RANK-SUM  TEST 


DOMAINS 

DIFFICULTY 

RANK 

Troubleshooting 

Electrical 

Equipment 

30 

1 

Electrical  Design 

35 

3 

Computer  Program 
Debugging 

3  1 

2 

Mission  Planning 

57 

4 

Gifted  Child 

Assessment 

63 

5 

Wr  ,ng  Research 
Proposals 

88 

7 

Medical  Diagnoses 

82 

6 

Musical  Performance 

98 

8 

Wl  =  1  +3+2+4 =  I  0 
W2  =  5*7*6*8  »  26 


C-  5 


'ABLE  a 


WILCOXGN  TWO-SAMPLE 
RANK-SUM  TEST 


DOMAINS 

NUM  OF 
EXCURSION 

RANK 

w  * 

Troubleshooting 

Electrical 

Equipment 

1  3 

2 

Electrical  Design 

i  6 

3 

Computer  Program 
Debugging 

4 

1 

Mission  Planning 

28 

4 

Sifted  Chi  Id 

Assessment 

30 

5 

Writing  Research 

Proposals 

3  1 

6 

Medical  Diagnoses 

88 

8 

Musical  Performance 

67 

7 

W  I  =  2*3*  1*4=  10 
W2  =  5*6*8*7  =  26 
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table  4 


WILCOXON  TWO-SAMPLE 
RANK-SUM  TEST 


DOMAINS 

CRITICAL 

VOLUME  RANK 

Troubleshooting 

Electrical 

35 1  2 

Equipment 

Electrical  Design 

304  4 

Computer  Program 

Debugging 

79 

Mission  Planning 

484  3 

0  if  teg  Chi  la 

Assessment 

384  5 

Writing  Research 

Proposals 

1031  6 

Medical  Diagnoses 

2546  7 

Musical  Performance 

2700  5 

W  1  =  2  *  4  ♦  i  +  3  =  10 

W2  =  5*6*7  +  8  -  26 
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Capabilities  of  Expert  Systems 
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5 . 5  CASE  STUDY 


In  order  to  demonstrate  the  use  of  the  ESDAAT,  a  case  study  was 
performed  on  a  domain  of  interest  which  had  as  yet  not  been  studied  for 
possible  implementation  of  an  expert  system.  The  domain  which  was 
selected  was  the  identification  of  points  on  computerized  topographies 
of  human  heads.  This  task  is  presently  performed  by  technicians  who 
work  at  the  Air  Force  Aeromedical  Research  Laboratory. 

5.5.1  MANAGEMENT 

The  management  of  the  Aeromedical  Laboratory  was  very  receptive  to 
using  the  ESDAAT  to  evaluate  the  suitability  of  developing  an  expert 
system  to  automate  the  identification  of  points  within  their  data.  At 
the  time  of  the  case  study,  the  management  was  considering  a  number  of 
AI  and  other  computer  analysis  techniques  to  automate  this  task.  There 
was  some  reluctance  to  commit  the  time  of  the  domain  experts  to  the 
evaluation  project.  However,  the  domain  experts  were  very  cooperative. 

5.5.2  DOMAIN  EXPERTS 

Two  domain  experts  filled  out  the  DSAT  questionnaire  for  the  domain 
of  point  by  picking  landmarks  from  the  computer  topography.  The  results 
of  the  DSAT  questionnaire  were  averaged  to  provide  the  raw  data  which 
was  fed  into  the  ESDAAT  analysis.  The  domain  experts  were  very 
enthusiastic  about  the  project.  It  is  clear  from  this  case  study  that 
knowledge  engineers  and  evaluators  will  need  to  spend  a  good  deal  of 
time  familiarizing  themselves  with  the  DSAT  survey  and  the  SOI  model 
before  they  can  effectively  administer  the  survey  to  domain  experts. 
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5.5.3  ANALYSIS  OF  DATA 


The  averaged  DSAT  inputs  were  run  in  the  level  one  and  level  two 
analysis  programs.  The  results  of  the  level  one  analysis  are  as 
follows : 

Probability  of  Success  60% 

Difficulty  56% 

Resources  Required  67% 

The  results  of  level  one  analysis  indicate  that  there  is  good 
reason  to  believe  that  an  expert  system  could  be  developed  for  this 
domain  with  at  least  some  level  of  success  if  enough  resources  are 
allocated  to  the  project. 

The  results  of  the  level  two  analysis  are  shown  in  Figure  15,  the 
requirements  chart;  Figure  16,  the  difference  chart;  and  Figure  17,  the 
listing  of  critical  elements  of  concern  in  the  dom  '  ..  As  can  be  seen 
in  these  figures,  there  are  a  number  of  elements  within  the  requirements 
of  the  domain  which  are  not  covered  by  expert  systems  capabilities. 
Therefore,  these  elements  of  the  domain  should  be  of  first  concern  when 
attempting  to  develop  an  actual  expert  system  in  the  domain. 

5.5.4  RECOMMENDATIONS  AND  ACCEPTANCE 

The  results  of  the  analysis  indicate  that  this  is  not  an  ideal 
domain  for  an  expert  system  solution.  However,  with  sufficient 
resources,  it  should  be  possible  to  develop  an  expert  system  to  automate 
some  part  of  the  domain.  The  results  of  the  level  one  analysis  were 
accepted  by  the  lab  management  with  little  need  of  additional 
explanation.  The  level  two  output  data  required  some  explanation,  both 
to  the  lab  management  and  to  the  designers  of  the  potential  expert 
system  who  were  unfamiliar  with  the  SOI  mode] .  As  a  result  of  the 
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ESDAAT  recommendation,  the  lab  is  proceeding  with  the  development  of  an 
expert  system  to  help  automate  the  point  picking. 
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DIVERGENT  PRODUCTION  SEMANTIC  CLASSES 
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6.0  DISCUSSION  AND  CONCLUSIONS 


6.1  EMPIRICAL  OBSERVATIONS 

6.1.1  USE  AND  USEFULNESS  OF  THE  TOOL 

Results  indicate  that  the  ESDAAT  can  give  decision  makers  and 
system  designers  a  great  deal  of  information  about  the  domain  in  which 
they  are  interested.  It  is  not,  however,  a  panacea.  The  ESDAAT  is  not 
designed  to  address  issues  other  than  the  suitability  of  the  domain  for 
an  expert  system  solution.  The  ESDAAT  will  not  design  a  system  for  an 
application  or  allocate  the  required  resources.  The  ESDAAT  will, 
however,  give  a  great  deal  of  information  that  can  guide  the  system 
designer  and  knowledge  engineer  to  appropriate  domains  for  building 
useful,  powerful  expert  systems. 

The  stated  objectives  of  this  project  as  listed  in  Section  4  of 
this  paper  were,  first,  to  demonstrate  the  validity  of  the  Guilford 
Structure  of  Intellect  model  for  mapping  characteristics  of  task 
domains.  Second,  to  develop  a  representation  of  the  existing 
capabilities  of  expert  systems  in  the  SOI  model.  And  third,  to  develop 
and  validate  mathematical  algorithms  for  comparing  domains  against  one 
another  and  against  expert  systems  capabilities. 

This  project  has  met  all  of  these  research  objectives.  In  Section 
5.4,  the  validation  of  the  model,  we  discuss  how  domain  systems  were 
selected  and  then  tested  in  order  to  demonstrate  that  the  SOI  model  can 
accurately  and  completely  represent  widely  varying  domains  in  a 
numerically  sensitive  manner.  We  successfully  derived  a  model  of 


83 


existing  expert  system  capabilities  which  is  presented  in  Section  5.1 
and  used  in  subsequent  analysis. 

We  have  also  developed  and  presented  in  this  paper  valid,  useful 
computer  algorithms  for  domain  selection  and  evaluation  for  expert 
systems. 

6.1.2  PATTERNS  IN  THE  LEVEL  TWO  ANALYSIS 

The  number  of  high  criticality  elements  in  a  domain  has  a 
significant  effect  on  the  difficulty  of  system  implementation.  Domains 
in  which  there  are  an  extreme  number  of  critical  elements  are  domains  of 
high  complexity.  These  domains  generally  cannot  be  implemented  as 
expert  systems.  However,  they  can  be  candidates  for  expert  systems 
which  address  some  subset  of  this  domain.  Domains  in  which  there  are 
large  numbers  of  difficult,  but  not  critical,  elements  are  generally 
domains  with  greater  flexibility  in  the  way  that  they  may  be  performed 
and/or  implemented  as  an  expert  system.  In  general,  domains  which  have 
fewer  elements  and  elements  with  lower  degrees  of  difficulty  are  easy  to 
implement.  Usually,  domains  of  interest  will  tend  more  to  complex 
domains  with  moderate  to  high  difficulty.  During  the  course  of  this 
research,  it  has  become  apparent  that  the  domain  analysis  can  also  give 
hints  and  direction  on  how  to  design  and  implement  an  expert  system. 

6.1.3  DOMAINS,  COMPLEXITY,  AND  EVALUATION 

One  of  the  most  interesting  things  we  learned  in  this  research  is 
that  domains  vary  in  complexity  in  ways  similar  to  the  way  we  describe 
levels  of  experts.  This  form  of  describing  domains  is  key  to  being  able 
to  determine  whether  expert  systems  can  or  cannot  be  designed  for 
dealing  with  an  entire  domain  or  whether  the  best  we  can  do  is  develop  a 
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solution  for  a  small  part  of  the  domain.  (Typically  an  expert  system 


domain  application  is  for  the  whole  domain. ) 

Just  as  there  are  different  levels  of  experts  (Novice,  Expert, 

Super  Expert),  there  are  different  levels  of  domain  size,  complexity, 
and  character. 

Size:  Resources  needed  to  implement  the  expert  system 

Character:  SOI  DSAT  modeling 

Complexity:  Toy  Domains  /  Level  0 

Sparse  Domains  /  Level  1 
Complex  Domains  /  Level  2 
Super  Domains  /  Level  3 

In  the  case  of  level  2  domains  (i.e.  complex  domains),  the 
difficulty  levels  increase  with  the  complexity  and  the  domain  will  most 
likely  require  super  experts  to  describe  it.  Also,  in  level  2  domains, 
it  is  likely  that  an  expert  system  will  not  be  able  to  solve  the 
complete  domain  task,  i.e.,  a  part  domain  implementation  will  be  more 
successful.  There  has  been  very  little  description  to  date  of  level 
three  domains.  Super  domains  are  characterized  by  being  so  complex  that 
humans  either  relegate  the  performance  of  the  domain  to  a  highly  elite 
subgroup  of  the  population  or  require  more  then  one  super  expert  to 
completely  perform  the  domain.  In  such  super  domains,  the  best  we  can 
hope  is  for  an  expert  system  to  be  implemented  for  some  small  subset  of 
the  super  domain.  An  example  of  a  super  domain  is  the  domain  of  medical 
diagnoses.  For  years  this  domain  has  required  many  super  experts,  each 
specializing  in  sub-domains  and  devoting  many  years  of  preparation  and 
study. 
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The  medical  domain  has  produced  a  number  of  good  expert  systems. 
These  have  been  successful  primarily  because  their  domain  has  been 
limited  to  a  small  sub-set  of  the  overall  super  domain. 

Once  a  level  2  or  3  domain  has  been  identified  as  a  domain  of 
interest  for  an  expert  system  implementation,  the  description  of  expert 
systems  capabilities  found  in  this  research  can  serve  as  a  guide  to 
selecting  a  suitable  subset  of  the  domain  for  which  to  implement  the 
expert  system. 

6.1.4  METHODOLOGICAL  PROBLEMS 

During  the  development  and  use  of  the  analysis  methods  for  the 
ESDAAT,  several  methodological  problems  were  noted  which  are  important 
in  using  the  tool  or  reproducing  the  results  reported  here.  The  first 
thing  that  is  important  to  note  is  that  the  administration  of  the  DSAT 
and  ESDAAT  questionnaires  requires  a  high  degree  of  training  and 
expertise  with  the  structure  of  intellect  model. 

The  next  most  important  aspect  of  the  methodology  and  procedure  is 
that  in  order  to  do  an  accurate,  orderly  analysis,  the  domain  experts 
must  have  a  clear,  well-defined  idea  of  the  domain  they  are  describing 
and  that  all  domain  experts  agree  exactly  as  to  the  nature  of  the  task. 
If  these  constraints  on  the  data  gathering  are  observed,  the  data 
analysis  will  yield  a  reliable  result. 

6.2  FUTURE  RESEARCH  AND  DEVELOPMENT 

6.2.1  SUMMARY 

The  development  of  the  ESDAAT  has  demonstrated  the  practicality  of 
theory-based  domain  suitability  analysis.  However,  there  are  a  number 
of  issues  which  have  arisen  during  this  research  which  require 
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additional  research  and  ongoing  study.  In  this  section  we  briefly 
address  a  number  of  thene  issues. 

6.2.2  CHANGES  IN  EXPERT  SYSTEMS  CAPABILITIES 

During  the  history  of  computer  sc  .mce,  in  general,  and  expert 
systems  in  particular,  there  have  been  changes  and  advancements  in  the 
understanding  of  the  capabilities  of  the  implementation  of  computer 
algorithms  to  solve  problems.  As  computer  hardware  and  software  design 
continues  to  develop,  these  capabilities  will  also  continue  to  increase. 
The  model  of  expert  system  capabilities  presented  here  should  not  change 
significantly.  This  conclusion  needs  to  be  researched  and  confirmed 
over  the  course  of  further  expert  systems  development. 

The  reason  we  believe  that  our  model  is  fundamentally  complete  is 
twofold:  (1)  the  fundamental  structure  of  expert  systems  implementation 

is  fully  defined  and  implemented  into  our  current  model,  and  (2)  there 
was  considerable  agreement  among  all  of  the  expert  knowledge  engineers 
as  to  the  capabilities  or  expert  systems. 

6.2.3  ISSUE  OF  DIFFICULTY  IN  TASKS 

One  of  the  areas  of  interest  in  evaluating  domain  suitability  must 
be  the  problem  of  trying  to  determine  the  level  of  difficulty  in  a 
particular  task  or  subtask  for  which  an  expert  system  can  be  designed  to 
perform  at  an  acceptable  level. 

What  are  the  consequences  of  the  level  of  difficulty  in  a 
particular  area  to  the  design  and  implementation  of  the  design  and 
implementation  of  the  expert  system? 
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6.2.4  AUTOMATION  OF  THE  ANALYSIS 


The  usability  and  full-scale  validation  and  evaluation  of  this 
design  resource  tool  will  require  the  development  of  an  automated 
version  of  the  DSAT  data  acquisition  tool  and  an  automation  of  this 
analysis. 

6.2.5  ISSUE  OF  KNOWLEDGE  ACQUISITION 

The  ESDAAT  deals  primarily  with  matching  the  characteristics  of  the 
domain  with  the  capabilities  of  expert  systems. 

However,  this  analysis  does  not  consider  in  large  part  one  of  the 
most  important  practical  issues  in  expert  systems  development,  that  of 
knowledge  acquisition  from  the  experts. 

6.2.6  TAILORING  EXPERT  SYSTEMS  DESIGN  BASED  ON  ESDAAT  DESCRIPTIONS 
During  the  course  of  this  research,  a  number  of  both  domain  experts 

and  system  developers  requested  that  the  output  of  the  ESDAAT  include 
information  about  what  approach  to  take  in  the  development  of  an  expert 
system  in  the  domain  of  interest.  The  basic  goals  and  theoretical 
construction  of  the  ESDAAT  do  not  allow  these  kinds  of  recommendations 
to  be  made  directly.  However,  there  are  indications  from  some  patterns 
in  the  level  two  analysis  that  this  information  may  be  available  in  some 
form.  In  the  next  iteration  of  the  ESDAAT  project,  this  issue  will  be 
investigated  further. 

6.3  CONCLUSIONS 

There  are  several  important  general  conclusion  to  be  drawn  from 
this  research.  First,  the  field  of  expert  systems  can  benefit  from  the 
inclusion  of  basic  principles  of  human  cognition  in  the  selection  of 
potential  application  domains.  Second,  if  properly  solicited,  domain 
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experts  can  and  will  provide  comprehensive  and  detailed  information 
regarding  the  nature  and  characteristics  of  the  domain  which  can  be  used 
to  determine  the  domain's  compatibility  with  expert  systems.  Third, 
theories  of  human  intelligence  form  a  reasonable  and  comprehensive 
modeling  space  in  which  to  describe  the  nature  and  characteristics  of 
domains  and  expert  systems  technologies.  Fourth,  an  analysis  tool  of 
the  characteristics  and  mappings  of  domains  will  provide  decision-makers 
with  a  useful  knowledge  of  the  domain  suitability  for  an  expert  systems 
technology  solution. 

There  are  also  several  important  specific  conclusions  to  be  drawn 
from  this  research.  The  results  described  in  Section  5  suggest  that  the 
Domain  Suitability  Analysis  Tool  is  ready  to  be  implemented  as  a  system 
analysis  tool  for  describing  domains  and  their  relative  suitability  to 
expert  systems  technology  solutions. 

The  representation  of  expert  systems  capabilities  presented  in  this 
research  is  an  important  step  forward  in  understanding  the  nature  of 
expert  systems  in  a  global  sense. 
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A.  STRUCTURED  INTERVIEW  WITH  KNOWLEDGE  ENGINEERS 


f 
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STRUCTURED  SURVEY  FOR  KNOWLEDGE  ENGINEERS 


Interviewer' s  Question  sheet 


I .  INTRODUCTION 


Explain  to  the  KE  the  general  purpose  of  this  project. 


There  are  no  theoretically  based,  empirically  established 
methods  for  determining  suitability  for  applying  rule-based  expert  systems 
technology.  While  several  ad  hoc  guidelines  have  been  proposed,  at  best,  they 
provide  only  general  guidelines  for  determining  if  a  domain  can  benefit  from 
rule-based  implementations  of  expert  systems  technology. 

The  goal  of  this  project  is  to  develop  A  methodology  which 
allows  the  knowledge  engineer  (  or  program  manager  )  to  accurately  determine 
the  degree  of  success  achievable  by  pursuing  an  expert  systems  technology 
solution  in  a  particular  problem  domain. 


Review  some  definition  of  the  terms  that  we  will  be  using 
(  domain, evaluation  tool,ect.) 


Make  note  of  the  recording  device (taping  the  interview  so  as  to 
better  retain  the  additional  comments.) 


Explain  the  DSAT  model  and  Jay  thesis  project. 

The  procedures  necessary  to  perform  an  accurate  evaluation  of 
potential  application  domains  for  expert  systems  technology  using  the 
theory-based  model  have  been  integrated  to  create  the  Domain  Suitability 
Analysis  Tool  (DSAT)  The  DSAT  has  been  designed  to  be  both  theoretically 
valid  and  operationally  efficient.  The  DSAT  is  designed  to  describe  the  domain 
information  elements  (based  on  the  SOI  model)  as  revealed  by  the  domain  expert 
and  then  produce  a  recommendation  on  the  support  provided  by  expert 
systems  technology  for  those  information  elements. 
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Explanation  of  the  Guilford,  Structure  of  Intellect  (SOI)  model. 


Show  the  Guilford  Cube  (figure  1  ) 


This  model  attempts  to  define  human  cognition  and  intelligence.  It  falls 
under  the  category  of  a  Psychometric  theory  of  intelligence.  Psychometric 
theory  is  based  on  separating  human  cognition  and  intelligence  into  different 
factors  or  elements. 

As  the  list  of  factors  began  to  mount,  several  similarities  and  differences 
became  apparent.  Some  factors  could  be  grouped  based  on  the  mental  processes 
involved,  such  as  cognition,  memory,  and  evaluation.  Others  could  be 
segregated  based  on  the  information  used;  symbolic,  semantic,  or  visual,  for 
example.  A  third  dimension  involved  the  form  of  the  information  used;  units, 
classes,  or  relations,  for  example. 

The  resulting  cubic  figure  that  encompasses  these  dimensions  is  the 
Structure-of-Intellect  (SOI)  model  (Guilford, 1967, 1985) .  The  figure  shows  the 
SOI  model  with  its  three  dimensions  :  operation,  contents,  and  products. 

This  model  illustrates  the  focus  of  psychometric  theories  on  cataloging  and 
systematically  identifying  the  components  of  intelligence  and  cognitive 
behavior . 
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EXPERIENCE  OF  THE  KNOWLEDGE  ENGINEER 


II  . 


A.  Domains 


1.  What  domains  (i.e.  what  kinds  of  problems/ fields)  are  you 
familar  with  working  with  in  the  development  of  expert 
systems? 

2.  What  similarities  do  you  see  in  problems  which  can  be 
addressed  by  expert  systems? 


3.  What  aspects  of  the  problem  domains  would  you  say  are 

the  most  important  to  address  in  order  to  assure  that  the 
expert  system  is  workable. 


B.  Expert  systems 


1 .  What  expert  systems  have  you  worked  on? 

Please  list  the  system  by  name  and  give  a  short 
description  of  the  system  and  how  well  it  performs. 
Include  systems  which  did  not  work  out  as  well  as 
successful  systems. 


2.  what,  if  any,  evaluation  tools  were  used  to  make 

decisions  about  whether  or  not  to  undertake  the  expert 
systems . 


C.  Areas  of  resonsibility 


Which  of  these  areas  have  you  participated  in  when 
developing  experts  systems. 

(  Please  note  the  area  and  give  a  short  explanation.) 


1.  Systems  feasibility  evaluation 
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.  Knowledge  elicitation 


3.  System  development  !  implementation 


Success  /  Failure 


1)  From  your  experience  what  about  a  problem  domain  would 
tend  to  indicate  to  you  that  an  expert  system  can  be 
developed  in  that  domain. 

2)  From  your  experience  what  about  a  problem  domain  would 
tend  to  indicate  to  you  that  an  expert  system  can  not 
be  developed  in  that  domain. 


E.  General 

1 .  Years  of  experience  total  in  technical  field. 

2.  Years  of  experience  in  working  with  expert  systems. 

3.  Educational  background 

a.  General 

b.  Expert  Systems 
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Description  of  come  expert  systems  wortced  on. 


(  For  eacn  of  the  Expert  Systems  discussed. ) 


Described  the  results  of  the  development  effort  in  terms  of: 


a.  likelihood  of  completicn/success 
o.  Difficulty 
c.  Resources 

1 )  K .  E  .  '  s 

2)  Domain  Experts 

:)  System  programmer 


Describe  or  relate  any  key  or  unique  lessons  learned  in  the  development 
of  these  expert  systems. 


Describe  as  test  as  you  can  the  problem  (domain)  for  which  this  expert 
system  was  designed  in  terms  of  the  Structure  of  Intellect  model. 


What  element  elements  within  the  model  would  require  the  greatest 
expenditure  of  resources  in  implementing. 


Knowledge  availability  is  very  important  to  the  development  of  expert 
systems.  What  elements  or  patterns  would  show  you,  as  a  knowledge 
engineer,  that  you  would  be  able  to  get  the  important  information  from 
the  domain  experts? 
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End  of  the  First  day.  . 


A) 

1.  Review  the  purpose  of  the  project. 

2.  Review  the  Structure  of  intellect  model. 

3.  Go  over  the  contents  of  the  second  part  of  the  survey. 


B) 


In  the  next  section  of  the  document  you  have  an  explanation  of  the 
contents  and  products  of  the  Structure  of  Intellect  model.  Please  look  over 
this  explanation  and  think  about  how  expert  systems  capabililes  could  best 
be  described  in  the  context  of  the  Structure  of  Intellect  model. 
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V. 


Description  of  Expert  systems  characteristics  in  terms  of  the 
Structure  of  intellect  model. 


(  Define  Buildability  as;  The  function  relating  the  domain  problem 
to  the  current  ability  of  expert  systems  to  perform  these  tasks) 


In  order  of  importance  rank  the  Operators  in  influence  over  the 
buildability  of  an  expert  system. 


In  order  of  importance  rank  the  Contents  influence  over  the 
buildability  of  an  expert  system. 


In  order  of  importance  rank  the  Products  influence  over  the 
buildability  of  an  expert  system. 


In  order  of  importance  rank  the  Dimension  elements  influence  over 
the  buildability  of  an  expert  system. 


How  would  you  describe  the  attributes  of  an  expert  system  which 
allow  it  to  solve  a  problem  in  a  given  domain  space? 


What  factors  would  you  say  are  the  most  deterministic  to  the  development 
of  an  expert  systems,  in  terms  of  the  Structure  of  Intellect  model? 


What  factors  would  you  say  are  most  important  to  determining  the 
difficulty  of  developing  a  system? 


What  factors  would  you  say  are  most  important  to  determining  the 
level  of  resources  needed  in  the  development  of  an  expert  system? 
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Analysis  of  the  Structure  of  Intellect  elements  in  the  developing 
an  Expert  System  (  Influence  of  Individual  elements  ) 


Going  down  the  list  of  individual  elements  of  the  SOI  model 
evaluate  each  element  on  a  scale  from  1-7  as  to  the  element's 
importance  to  the  difficulty  of  implementing  an  expert  system. 

Mote;  7  being  the  most  important  ,  1  having  r.o  influence  on  the 

problem  or  system. 

(see  answer  sheet) 


Analysis  of  the  ability  of  an  expert  system  to  perform  tasks  from 
the  SOI  model 


Going  down  the  list  of  individual  elements  of  the  SOI  model, 
evaluate  each  element  on  a  scale  from  1  to  7  as  to  whether  an 
expert  system  could  easily  perform  this  kind  of  task. 

Note;  7  being  difficult  to  perform 
1  being  very  easy  to  perform 


(see  answer  sheet) 


VII  . 


Other  Issues 


I.  Desired  output 


a.  What  information  would  be  of  most  use  to  you  in  evaluating 
a  domain  or  problem  for  an  expert  system  development? 


b.  In  what  format  would  you  prefer  to  see  this  information  presented? 


2.  Critical  Element  Analysis 


3.  Flexibility 


4.  Additional  Comments  on  Expert  Systems: 


5.  Additional  Comments  on  this  project: 


6.  Additional  Comments  on  this  survey: 
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KNOWLEDGE  ENGINEER  SURVEY 


SECTION  I  INTRODUCTION 


In  this  survey  you  will  be  asked  to  evaluate  the  capabilities 
of  Expert  systems  to  solve  problems. 

The  way  in  which  you  will  evaluate  these  capabilities  is  to 
evaluate  the  ability  of  expert  systems  to  perform  given 
kinds  of  tasks. 

These  tasks  are  defined  by  Guilford  in  his  Structure  of  Intellect 
model.  This  model  is  used  to  describe  elements  of  operator 
task  domains. 


See  Figure  1  The  Structure  of  Intellect  Cube 


SECTION  II  EXPERIENCE  OF  THE  KNOWLEDGE  ENGINEER 


A.  Domains 

1 . 


2.  Similarities 
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3.  Aspects 


Expert  Systems 


C.  Areas  of  Responsibility 


D.  Successes  /  Failure 


E.  General 


3a. 
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SECTION  III  DESCRIPTION  OF  SOME  EXPERT  SYSTEMS  WORKED 

la . 

lb. 

lc . 

1) 

2) 

3) 
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Explanation  /  Examples  to  describe  the  elements  of  the  SOI  model. 

In  the  following  sections  there  is  an  explanation  of  the  contents  and 
products  of  the  Structure  of  Intellect  model.  As  you  read  over  these  examples 
and  explanations,  try  to  think  about  how  these  elements  relate  to  the  ability 
of  an  expert  system  to  solve  problems  in  a  specific  domain  which  includes  the 
sections  of  the  SOI  model  which  is  being  explained. 

Please  feel  free  to  make  notations  in  your  sheets  that  might  help  you 
answer  questions  later. 
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SECTION  IV.  DESCRIPTION  OF  EXPERT  SYSTEMS  CHARACTERISTICS  IN  TERMS  OF 
THE  STRUCTURE  OF  INTELLECT  MODEL 


STRUCTURE  OF  INTELLECT  DOMAIN 


1)  Operators 


a . 

Cognition 

1 

b . 

Memory 

Divergent  Production 

*3 

d. 

Convergent  Production 

4 

fa  . 

Evaiuation 

5 

2)  Contents 


a.  Visual  1 

b.  Auditory  2 

c.  Symbolic  3 

d.  Semantic  4 

e.  Behavioral  5 


3)  Products 


a.  Units  1 

b.  Classes  2 

c.  Relations  3 

d.  Systems  4 

e.  Transformations  5 

f.  Implications  6 
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Dimensions 


F  requency 
Criticality 
*  Difficulty 
Flexibility 


SECTION  V.  ANALYSIS  OF  THE  STRUCTURE  OF  STRUCTURE  ELEMENTS  IN  DEVELOPING 

AN  EXPERT  SYSTEM  (INFLUENCE  IF  INDIVIDUAL  ELEMENTS) 


Elements  Importance 

VISUAL  CONTENTS  _ 

Visual  Units  _ _ 

Visual  Classes  _ _ 

Visual  Relations  _ 

Visual  Systems  _ 

Visual  Transformations  _ _ 

Visual  Implications  _ 

AUDITORY  CONTENTS  _ 

Auditory  Unit3  _ 

Auditory  Classes  _ 

Auditory  Relations  _ 

Auditory  Systems  _ 

Auditory  Transformations  _ 

Auditory  Implications  _ 


SYMBOLIC  CONTENTS 


Symbolic  Units 
Symbolic  Classes 
Svmoolic  Relations 
Symbolic  Systems 
Symbolic  Transformations 
Symbolic  Implication 


SEMANTIC  CONTENTS 
Semantic  Units 
Semantic  Classes 
Semantic  Relations 
Semantic  Systems 
Semantic  Transformations 
Semantic  Implications 

EEHAVIORAL  CONTENTS 
Behavioral  Units 
Behavioral  Classes 
Behavioral  Relations 
3ehavioral  Systems 
Behavioral  Transformations 
Behavioral  Implications 


SECTION  VI.  ANALYSIS  OF  ABILITY  OF  EXPERT  SYSTEMS  TO  PERFORM  TASKS 
FORM  THE  SOI  MODEL 
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Elements 


Difficulty 


VISUAL  CONTENTS 
Visual  Units 
Visual  Classes 
Visual  Relations 
Visual  Systems 
Visual  Transformations 
Visual  Implications 


AUDITORY  CONTENTS 
Auditory  Units 
Auditory  Classes 
Auditory  Relations 
Auditory  Systems 
Auditory  Transformations 
Auditory  Implications 

SYMBOLIC  CONTENTS 
Symbolic  Units 
Symbolic  Classes 
Symbolic  Relations 
Symbolic  Systems 
Symbolic  Transformations 
Symbolic  Implication 


SEMANTIC  CONTENTS 
Semantic  Units 
Semantic  Classes 


Semantic  Relations 
Semantic  Systems 
Semantic  Transformations 


Semantic  Implications 


BEHAVIORAL  CONTENTS 
Behavioral  Units 
Behavioral  Classes 
Behavioral  Relations 
Behavioral  Systems 
Behavioral  Transformations 
Behavioral  Implications 


SECTION  VII.  OTHER  ISSUES 


la. 


lb. 


2. 
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3. 


4.  Comments: 


5.  Comments: 


6.  Comments: 


117 


The  Structure  of  Intellect  Cube  (Guilford,  1967;  1985). 
F  1  a  ure  J, 
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A.l  RESULTS  OF  KNOWLEDGE  ENGINEER  SURVEY 
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SECTION  IV.  DESCRIPTION  OF  EXPERT  SYSTEMS  CHARACTERISTICS  IN  TERMS  OF 
THE  STRUCTURE  OF  INTELLECT  MODEL 


STRUCTURE  OF  INTELLECT  DOMAIN 


1)  Operators 


a.  Cognition  1  d 

b.  Memory  2  b 

c.  Divergent  Production  3  a 

d.  Convergent  Production  4  e 

e.  Evaluation  5  c 


2)  Contents 


a.  visual  1  c 

b.  Auditory  2  d 

c.  Symbolic  3  e 

d.  Semantic  4  a 

e.  Behavioral  5  ^ 


3)  Products 


a.  Units  1  b 

b.  Classes  2  a 

c.  Relations  3  ^ 

d.  Systems  4  c 

e.  Transformations  5  e 

f .  Implications  6  f 
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Dimensions 


Frequency 

Criticality 

Difficulty 

Flexibility 


SECTION  V.  ANALYSIS  OF  THE  STRUCTURE  OF  STRUCTURE  ELEMENTS  IN  DEVELOPING 

AN  EXPERT  SYSTEM  (INFLUENCE  IF  INDIVIDUAL  ELEMENTS) 


Elements  Importance 

VISUAL  CONTENTS  _ 1 _ 

Visual  Units  1 

Visual  Classes  1 

Visual  Relations  1 

Visual  Systems  1 

Visual  Transformations  1 

Visual  Implications  1 


AUDITORY  CONTENTS  2 

Auditory  Units  j. 

Auditory  Classes  2 

Auditory  Relations  1 

Auditory  Systems  1 

Auditory  Transformations  2 

Auditory  Implications  2 


SYMBOLIC  CONTENTS 
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Symbolic  Units  6 

Symbolic  Classes  6 

Symbolic  Relations  7 

Symbolic  Systems  6 

Symbolic  Transformations  5 

Symbolic  Implication  6 


SEMANTIC  CONTENTS  _ 6 _ 

Semantic  Units  4 _ 

Semantic  Classes  3 _ 

Semantic  Relations  5 _ 

Semantic  Systems  6 

Semantic  Transformations  7 _ 

Semantic  Implications  7 _ 

BEHAVIORAL  CONTENTS  _ 2 _ 

Behavioral  Units  1 

Behavioral  Classes  2 

Behavioral  Relations  2 

Behavioral  Systems  1 

Behavioral  Transformations  2 

Behavioral  Implications  1 

SECTION  VI.  ANALYSIS  OF  ABILITY  OF  EXPERT  SYSTEMS  TO  PERFORM  TASKS 
FORM  THE  SOI  MODEL 
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Elements 


Difficulty 


VISUAL  CONTENTS  _ 6 

Visual  Units  5 
Visual  Classes  7 
Visual  Relations  6 
Visual  Systems  7 
Visual  Transformations  7 
Visual  Implications  6 


AUDITORY  CONTENTS  _ 7 

Auditory  Units  7 
Auditory  Classes  7 
Auditory  Relations  6 
Auditory  Systems  7 
Auditory  Transformations  7 
Auditory  Implications  6 


SYMBOLIC  CONTENTS  ! 

Symbolic  Units  1 

Symbolic  Classes  1 

Symbolic  Relations  1 

Symbolic  Systems  1 

Symbolic  Transformations  2 

Symbolic  Implication  1 


SEMANTIC  CONTENTS  g 

Semantic  Units  4 

Semantic  Classes  5 

Semantic  Relations  4 

Semantic  Systems  7 

Semantic  Transformations 
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7 


Semantic  Implications  7 


BEHAVIORAL  CONTENTS  7 

Behavioral  Units  7 

Behavioral  Classes  7 

Behavioral  Relations  6 

Behavioral  Systems  6 

Behavioral  Transformations  7 

Behavioral  Implications  7 
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B.  CRITICAL  ELEMENT  SURVEY 
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CRITICAL  ELEMENT  SUR  Vfp 


The  purpose  of  this  questionnaire  is  to  obtain  additional  information  about 
some  key  parts  of  the  model  of  the  domain  which  we  are  studying. 

For  each  of  the  Critical  Elements  identified  by  the  excursion  analysis 
algorithm  please  answer  the  following  questions. 

When  answering  these  questions  please  try  to  limit  your  responses  to  the 
one  part  of  the  model  that  we  are  investigating  at  that  time. 


What  about  this  part  of  the  overall  domain  task  makes  it  important  in 
successfully  completing  the  domain  task? 
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A . 1  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  likelyhood  of  success? 


A. 2  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  difficulty  of  the  domain? 


A. 3  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  resources  required? 


B.l  How  critical  is  this  part  of  the  domain  to  the  overall  success  of  the 
project  is  this  part  of  the  task? 


B.2  Flexibility:  can  this  part  of  the  domain  be  implemented  or  performed  in 
some  other  way?  If  it  can  please  describe  in  what  other  part  of  the  model  the 
task  could  be  preformed? 
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B.3  How  difficult  is  this  part  of  the  domain  to  preform? 


3.4  How  frequently  is  this  part  of  the  domain  performed  in  the  overall 

task? 


C.l  Is  there  some  agreement  on  how  to  do  this  ta3k? 


C.2  Is  there  some  level  of  agreement  on  what  is  a  good  solution  to  the  domain 

problems? 
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What  level  of  expertise  is  required  to  be  a  ture  expert  or  master  in 


C.  3 

this  domain? 


C.4  What  effect  does  this  part  of  the  overall  domain  task  have  on  the 
ability  of  the  systems  designers  to  obtain  information  about  the  domain? 


C . 5  What  effect  does  this  part  of  the  over  all  domain  ta3k  have  on  the 

complexity  of  modeling  or  describing  the  over  all  task? 


C . 6  Is  this  part  of  the  domain  well  defined? 


D.l  Assuming  That  This  part  of  the  domain  is  difficult  for  an  expert  system 
to  perform  how  difficult  is  this  part  of  the  domain  for  the  human  expert  to 
perform? 
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D.2  If  an  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  expert  in  performing  in 
this  domain? 


D. 3  If  an  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  Novice? 


D.4  Do  you  think  you  would  have  more  confidence  in  the  usability/performance 
of  an  expert  system  of  thi3  part  of  the  domain  was  addressed  by  the  expert 
system?  Why. 


132 


B.l  FIRST  PROTOTYPE 


Results  of  the  First  prototype  of  the  Critical  Element  Survey 

Domain  is  Mission  Planning 

Element  is  Cognition  of  Behavioral  Units 


CRITICAL  ELEMENT  SURVEY 


The  purpose  of  this  questionnaire  is  to 
some  key  parts  of  the  model  of  the  domain 


obtain  additional  information 
which  we  are  studying. 


about 


For  each  of  the  Critical  Elements  identified  by  the  excursion  analysis 
algorithm  please  answer  the  following  questions. 


When  answering  these  questions  please  try  to  limit  your  responses  to  the 
one  part  of  the  model  that  we  are  investigating  at  that  time. 


What  about  this  part  of  the  overall  domain  task  makes  it  important  in 
successfully  completing  the  domain  task? 


Ans .  Subject  would  miss  or  unintentionally  read  an  observed  expression  and 
misinterpret  the  data.  This  could  drive  the  domain  expert  to  include  or 
exclude  pertinent  or  questionable  data  thereby  skewing  the  outcome. 
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A . 1  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  likelyhood  of  success? 

Ans .  If  recognized,  this  error  can  be  corrected. 


A. 2  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  difficulty  of  the  domain? 

Ans.  Can  influence  systems  solutions  on  rare  occasions 


A. 3  what  i3  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  resources  required? 


Ans.  Can  influence  the  whole  span  of  the  solution. 


B.l  How  critical  is  this  part  of  the  domain  to  the  overall  success  of  the 
project  is  thi3  part  of  the  task? 


Ans.  Possibility  exists,  that  if  (1)  unrecognized  and  given  a  high 
probability  of  happening  against  a  critical  event.  Then  the  outcome  (2)  could 
be  critical.  Otherwise,  as  an  exact  opposite  to  this  case. 


B.2  Flexibility:  can  this  part  of  the  domain  be  implemented  or  performed  in 
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some  other  way?  If  it  can  please  describe  in  what  other  part  of  the  model  the 
task  could  be  preformed? 

Ans .  Yes,  verbally 


B.3  How  difficult  is  this  part  of  the  domain  to  preform? 


Ans .  Easy 


B.4  How  frequently  is  this  part  of  the  domain  performed  in  the  overall 

ta3k? 

Ans.  Infrequently 


C.l  I 3  there  some  agreement  on  how  to  do  this  task? 


Ans.  Yes 


C.2  Is  there  some  level  of  agreement  on  what  is  a  good  solution  to  the  domain 

problems? 


Ans.  No 
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c. 3  What  level  of  expertise  Is  required  to  be  a  true  expert  or  master  in 

this  domain? 

Ans .  Moderate 


C.4  What  effect  does  this  part  of  the  overall  domain  task  have  on  the 
ability  of  the  systems  designers  to  obtain  information  about  the  domain? 

Ans.  Minimal 


C.5  What  effect  does  this  part  of  the  over  all  domain  task  have  on  the 

complexity  of  modeling  or  describing  the  over  all  task? 

Ans.  Possibly  no  effect 


C.6  Is  this  part  of  the  domain  well  defined? 

Ans .  No 


to  perform  how  difficult  is  this  part  of  the  domain  for  the  human  expert  to 
perform? 

Ans.  Easy  to  perform,  yet  easy  to  misinterpret. 


D.2  If  an  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  expert  in  performing  in 
this  domain? 

Ans .  Yes 


D . 3  If  an  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  Novice? 


Ans.  Yes 


D.4  Do  you  think  you  would  have  more  confidence  in  the  usability/performance 
of  an  expert  system  if  this  part  of  the  domain  was  addressed  by  the  expert 
system?  Why. 

Ans.  No,  machines  don't  smile  or  frown. 
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B.2  SECOND  PROTOTYPE 
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Results  of  the  Second  prototype  of  the  Critical  Element  Survey 
Domain  is  Mission  Planning 

Element  is  Cognition  of  Semantic  Implications 

CRITICAL  ELEMENT  SURVEY 


The  purpose  of  this  questionnaire  is  to  obtain  additional  information  about 
some  key  parts  of  the  model  of  the  domain  which  we  are  studying. 

For  each  of  the  Critical  Elements  identified  by  the  excursion  analysis 
algorithm  please  answer  the  following  questions. 

When  answering  these  questions  please  try  to  limit  your  responses  to  the 
one  part  of  the  model  that  we  are  investigating  at  that  time. 


What  about  this  part  of  the  overall  domain  task  makes  it  important  in 
successfully  completing  the  domain  task? 

Ans.  Symbology  plays  a  major  role  in  this  task  since  recognizing, 
interpreting  and  using  symbols  is  a  basic  skill  required  to  perform  this  task. 
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A.l  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  likelyhood  of  success? 


Ans .  Fairly  Significant 


A. 2  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  difficulty  of  the  domain? 


Ans.  Fairly  significant 


A. 3  What  is  the  influence  of  this  part  of  the  domain  over  the  systems 
solution  in  terms  of  the  resources  required? 


Ans.  Little  influence 


B.l  How  critical  is  this  part  of  the  domain  to  the  overall  success  of  the 
project  is  this  part  of  the  task? 

Ans.  Fairly  significant 
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B . 2  Flexibility:  can  this  part  of  the  domain  be  implemented  or  performed  in 

some  other  way?  If  it  can  please  describe  in  what  other  part  of  the  model  the 


task  could  be  preformed? 

Ans.  Inflexible 


B.3  How  difficult  is  this  part  of  the  domain  to  preform? 

Ans.  Fairly  simple 


B . 4  How  frequently  is  this  part  of  the  domain  performed  in  the  overall 

task? 

Ans.  Infrequently 


C.l  I 3  there  some  agreement  on  how  to  do  this  task? 

Ans.  Well  defined 


C.2  Is  there  some  level  of  agreement  on  what  is  a  good  solution  to  the  domain 

problems? 
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Ans.  Yes 


C.3  What  level  of  expertise  is  required  to  be  a  ture  expert  or  master  in 
this  domain? 


Ans.  Beginner  to  moderate 


3.4  What  effect  does  this  part  of  the  overall  domain  task  have  on  the 
ability  of  the  systems  designers  to  obtain  information  about  the  domain? 


Ans.  Minimal 


C.5  What  effect  does  this  part  of  the  over  all  domain  task  have  on  the 
complexity  of  modeling  or  describing  the  over  all  task? 


Ans.  My  guess  is  minimal  although  I  unsure  of  a  systems  ability  to  perform 
this  task. 


C.6  Is  this  part  of  the  domain  well  defined? 


Ans .  Yes 
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D.l  Assuming  That  This  part  of  the  domain  is  difficult  for  an  expert  system 
to  perform  how  difficult  is  this  part  of  the  domain  for  the  human  expert  to 
perform? 


Ans .  Fairly  simple 


D.2  If  an  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  expert  in  performing  in 
this  domain? 


Ans.  Possibly,  Domain  expert  would  need  to  use  work  around  procedures  to 
modify  ir  correct  errors  made  by  expert  system. 


D.3  If  ’n  expert  system  where  to  be  developed  with  this  part  of  the  domain 
not  addressed  would  it  still  be  useful  to  a  domain  Novice? 


Ans.  Yes,  same  reason  as  D.2 


D.4  Do  you  think  you  would  have  more  confidence  in  the  usability/perf ormance 
of  an  expert  system  of  this  part  of  the  domain  was  addressed  by  the  expert 
system?  Why. 


Ans.  Yes,  Successful  recognition  and  implementation  of  this  part  of  the 
domain  is  critical  to  the  overall  success  of  this  portion  of  the  task. 
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C.  DATA  ANALYSIS 
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C.l  REQUIREMENT  CHARTS 
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Requirments  Medical  Dx.  Domain 


Requirments  Musical  Performance  Domain 


Units  Visual  Auditory  Symbol  Semantic  Behavioral 

Classes 
Relation  s 

Transformations  Contents  &  Products 


Requirments  Point  Picking  Domain 


C . 2  DIFFERENCE  CHARTS 
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Listing  of  Elements  where  Domain  Req.  are  greater  then  E.  S.  Capabilities 


o 

03 

in 


0'ccDoo\iri»ujMMOix)co>ja')jijiw 


r 

w 


o  o  o  a>  a»  i 


ooc»(j\ovO\o\a^»o\Aw  w 


«CDCoaCD(B(BOO\^0»9\0\>f>»AOo9i 


> 

r 


o 

G 


pi  m 
<  < 
>  > 
r  r 
a  a 
>  > 
H  *-3 

r-i  w 

O  O 

Z  Z 


n  pi 

o  < 
o  > 
z  g 

M  C 
>•3  > 
m  H 
O  w 
Z  O 
Z 


P 


C 

5T 


K  X  O  X  SC 

IP  >  M  >  > 

<  <  H  <  < 

>-*  M  O  *-*  »-» 

o  o  »  o  o 

W  50  K  w  5d 

>  >  >  >  .  . 

r  r  r  r  f  r 


n 

n 

o 

PI 

O 

O 

D 

D 

O 

o 

n 

D 

O 

O 

1 

o 

o 

o 

< 

M 

H 

H* 

M 

M 

o 

o 

M 

n 

*o 

J 

o 

a 

o 

Q 

> 

< 

< 

< 

< 

< 

o 

o 

< 

< 

PI 

l 

0 

z 

z 

z 

r* 

n 

PJ 

PI 

PI 

m 

z 

z 

PI 

PI 

50 

1 

a 

M 

M 

M 

c 

w 

50 

50 

50 

V 

»-4 

M 

50 

50 

> 

1 

e* 

H 

H 

-3 

> 

Q 

a 

O 

a 

a 

H 

G 

a 

►J 

l 

M 

M 

M 

-3 

PI 

n 

PI 

pi 

pi 

M 

*H 

PI 

pi 

C 

1 

o 

o 

O 

M 

z 

z 

z 

z 

z 

O 

o 

Z 

z 

50 

z 

z 

z 

O 

H 

H 

H 

H 

z 

z 

H 

H 

o 

50 

z 

T3 

•O 

•0 

T3 

•o 

*o 

50 

50 

50 

50 

» 

W 

50 

• 

o 

O 

O 

O 

O 

o 

O 

ll 

a 

O 

a 

o 

a 

o 

o 

►- 

» 

G 

G 

G 

G 

G 

G 

G 

o 

n 

n 

n 

n 

n 

o 

n 

® 

H 

H 

H 

►3 

H 

•s 

*■3 

l 

►-« 

M 

M 

H 

M 

M 

M 

1 

o 

o 

o 

o 

o 

o 

o 

l 

n 

z 

z 

z 

z 

z 

z 

z 

I 

0 

1 

l 

1 

rt 

0 

n 

03 

CD 

> 

> 

> 

> 

< 

> 

< 

> 

> 

> 

> 

O 

i 

pj 

n 

G 

G 

G 

c 

M 

c 

M 

G 

c 

G 

G 

O 

i 

r* 

X 

X 

O 

O 

O 

o 

to 

o 

to 

a 

o 

D 

a 

z 

I 

or 

> 

> 

M 

H 

M 

M 

G 

M 

c 

n 

M 

M 

M 

•-3 

I 

0 

< 

< 

H 

H 

H 

H 

> 

•-3 

> 

H 

►3 

H 

H 

PI 

l 

0 

H 

M 

O 

O 

O 

o 

r* 

O 

t4 

O 

O 

O 

O 

Z 

1 

o 

O 

» 

50 

50 

50 

50 

50 

50 

» 

50 

H 

1 

PI 

X 

50 

K 

K 

K 

K 

«: 

K 

K 

to 

1 

MnncMnccMHww»»dMwr> 

srrzzrzzzatKxwwZKr* 

r|MinHr,oiHHr,c-,HH>>r,H(A 
HWWWHI/lMWHNnnHH 
n  pi  pi  ~  r? 

>  to  to  >  in 

►3  -3  H  H 


n  n  z  at  m  w  , , 

>>intoOO>toto 


m  in 

n  3  pi 


z  z 

in  to 


O 

z 

to 


o  o  o  o 

z  z  z  z 

to  to  to  to 


T3 

» 

O 

O 

c 

o 

H 

to 


n 

•o 

Bt 

O’ 


rt 

y- 

0 

U) 


173 


Diff  (Medical  Dx .  Doaain  -  Expert  Systeas)  15:09  Wednesday,  August  7,  1991 


■b  KJ  CD  O 


o  o  o  n 

H»  HI  M  O 

<  <  <  o 
pi  w  pi  a 

?J  »  »  H 
O  O  O  H 
n  n  n  m 

z  a  z  o 
-i  H  H  z 

W  W  W 
WWW 
O  O  O 
d  O  O 
C  G  C 

non 

-*  H  H 

N  H  M 

o  o  o 

z  z  z 


®  CD  W  CJJ 
PI  PI  PI  PI 
X  X  3  X 
>  >  >  > 
<  <  a  < 

H  H  M  H 

O  O  HI  o 
w  w  n  w 
>  >  > 
r*  r  r 


n 

w 


H 

*-i 

> 


O 

c 


> 

H 

O 


o 

z 

PI 

z 

-J 


K  W  »  3 
<*  r  r  w 
h  >  >  r 

n  H  h  h 
1  h  m  n 
V»  O  O  > 
z  a  h 

M  l/I  H 

o 

z 

w 


w 

w 

o 

o 

c 

n 

H 


^  >  Ol  *  UJ  I 


M 

> 


o  O  O  A  U>  3\ 


n  z  x  n  x  x 

O  Pi  Pi  o  pi  PI 
2  2  2  2  2  3 
<  o  o  <  o  o 
<  W  PI  w  w 
:  k 


W  *5 

K  W 

o 

o 

PI 

PI 

z 

z 

H 

w 

w 

w 

w 

o 

o 

o 

o 

G 

G 

n 

n 

M 

o 

o 

z 

z 

w 

> 

H 

O 


o 

*"*  a 

•“O'oavj^uiAWMH  c/j 


OOOOOOO00000V 


^pO\flDwe8<7i(n^o,m 


a  a  n  n  n  o  . .  . . 
oooooooo 

ZOOOOOOCl 

<ZZZ2Z22 

COl^HtHIMHlHIM 

WHHHHHHH 

QhiMHIHIMhIHI 

PIOOOOOOO 

zzzzzzzz 

H 


*n 


o 

o 

G 

o 


o  n  n  o  n 
o  o  o 
Z  Z  Cl 

<  <  z 

PI  PI  M 

W  W  H 

o  a  i 

PI  PI 

z  z 
H 

w  *n 
w  w 
O  o 
o  o 

G  G 
r)  n 

W  H 


o 

;  z 


n 

w 


> 

r 


o 

G 


W 

> 

H 

O 

w 


o  t/»  o  x  3  ; 

>•<  c  HI  >  > 

H  >  H  <  Z 

o  r*  o  m 

w  wo 

•<  k  w 

> 


r  >  r  p*  - 

»  z  >  > 


o  o  o  o  o 
z  »  z  z  w 

M  2  M  M 
> 

H  H 


o 

z 


o 

z 


o 

z 


c 

sr 


w 

ii 


w 

«Q 


1 

o 

O 

o 

l 

•A 

1 

z 

z 

z 

1 

1 

1 

• 

1 

* 

1 

1 

r* 

v» 

n  i 

v> 

<  <;  <  w  < 

CD 

CD 

03 

co 

10 

n 

1 

1 

9 

pi 

a  i 

Pi 

H  M  M  PJ  H 

PI 

PI 

PI 

PI 

PI 

o 

I 

2 

Z  1 

2 

M  Ifl  W  I  01 

X 

X 

X 

X 

3 

z 

1 

r* 

> 

HI  1 

> 

C  C  G  >  C 

> 

> 

> 

> 

> 

Hj 

I 

sr 

z 

PI  1 

Z 

>  >  >  <  > 

< 

< 

< 

< 

z 

PJ 

1 

• 

H 

Z  1 

r| 

r  t-  r»  m  f 

HI 

M 

HI 

*-* 

H 

z 

1 

9 

M 

«H  1 

HI 

o 

o 

o 

o 

o 

M 

1 

n 

(A  1 

n 

w 

w 

w 

w 

w 

o 

V) 

1 

PI 

1 

> 

> 

> 

> 

> 

i 

1 

\ 

1 

| 

i r 

r 

r 

r 

r 

1 

1 

1 

V> 

» 

t 

«0  1 

M 

W  H  W  M  * 

o 

c 

n 

c 

w 

"0 

1 

1 

I 

o 

• 

pi 

W  1 

3 

2  M  >4  Pi  PI 

r 

z 

r 

z 

PI 

W 

1 

■a 

O  I 

w 

w  >  w  t-»  r 

> 

M 

> 

HI 

r* 

o 

1 

• 

> 

O  1 

r 

r  a  *-j  >  > 

to 

H 

w 

H 

> 

a 

1 

Xf 

G  1 

w 

h  In  «  H  H 

</> 

W 

VI 

V) 

G 

1 

H 

*■* 

n  i 

n 

ni  jmh 

pi 

pi 

HI 

n 

I 

H» 

o 

*4  1 

> 

>  O  (/>  o  o 

w 

w 

o 

H 

1 

H- 

z 

VI  1 

**  »  z  z 

z 

W 

1 

n 

VI 

i 

H 

h  2  In  in 

c n 

t 

*•' 

i 

a 

o  > 

1 

• 

i 

z 

z  ►* 

1 

01 

i 

VI 

tft  HI 

1 

O 

o 


174 


Expert  systems)  15:21  Thursday,  June  20,  1991 


j>i&UWWWUlWUUUWMOKjrOKJMKJI 


OOOOOOOOOOOOOOOl 


CO  CD  OB  CD 


>«kOOoooocnooooooo»o%^oiot 


o 

X 


> 

r 


X 

X 

n 

o 

c 


3 

<a 


PI 

n 

n 

n 

O 

n 

n 

n 

o 

a 

3 

3 

3 

3 

n 

w 

to 

n 

3 

n 

o 

1 

rt 

< 

< 

< 

o 

o 

o 

o 

o 

M 

pi 

pi 

Pi 

n 

pi 

o 

< 

< 

o 

PI 

o 

•0 

1 

ft 

> 

> 

> 

2 

a 

a 

a 

a 

< 

3 

3 

3 

3 

3 

Cl 

> 

> 

a 

3 

Cl 

PI 

1 

r 

r 

r 

< 

< 

< 

< 

< 

pi 

O 

O 

O 

O 

O 

a 

p* 

p* 

< 

O 

a 

JO 

1 

o 

G 

c 

c 

m 

pi 

n 

n 

pi 

JO 

JO 

JO 

50 

JO 

X 

M 

G 

G 

pi 

JO 

M 

> 

1 

o 

> 

> 

> 

X 

w 

» 

X 

x 

o 

K 

K 

K 

K 

K 

X 

> 

> 

50 

K 

X 

X 

1 

■ 

X 

H 

H 

ci 

ci 

<n 

O 

o 

pi 

t-M 

X 

X 

Cl 

H 

o 

1 

» 

M 

M 

PI 

pi 

w 

n 

PI 

a 

o 

H 

M 

PI 

o 

50 

J 

H- 

o 

o 

O 

z 

a 

a 

a 

a 

x 

a 

o 

o 

a 

a 

1 

3 

25 

a 

2 

X 

H 

H 

H 

H 

a 

a 

H 

X 

O 

50 

*o 

X 

X 

X 

x 

50 

x 

►H 

• 

50 

X 

X 

JO 

JO 

O 

50 

X 

U3 

O 

O 

o 

O 

o 

o 

O 

•n 

a 

a 

o 

o 

o 

o 

G 

O 

a 

c 

c 

c 

c 

G 

r> 

G 

• 

n 

n 

n 

n 

n 

X 

n 

• 

H 

X 

X 

X 

X 

X 

M 

X 

-j 

• 

Kt 

M 

M 

M 

M 

O 

M 

o 

O 

O 

o 

O 

a 

o 

i 

-a 

2 

a 

a 

a 

a 

a 

i 

00 

00 

> 

> 

> 

> 

> 

(A 

> 

> 

> 

> 

> 

> 

tA 

01 

00 

(A 

tA 

pi 

PI 

G 

G 

G 

G 

C 

G 

PI 

c 

c 

c 

G 

G 

c 

PI 

PI 

PI 

PI 

PI 

n 

X 

X 

O 

0 

O 

O 

a 

O 

3 

o 

o 

o 

O 

o 

o 

3 

3 

X 

3 

3 

o 

>  > 

M 

M 

M 

M 

►4 

> 

M 

►4 

H 

►4 

w 

> 

y 

y 

> 

> 

a 

< 

< 

X 

►1 

X 

H 

H 

HI 

a 

H 

X 

X 

X 

X 

X 

a 

a 

< 

a 

a 

X 

M 

w 

O 

0 

O 

O 

O 

O 

X 

O 

o 

o 

o 

o 

o 

X 

X 

M 

X 

X 

pi 

o 

O 

X 

X 

X 

X 

X 

X 

M 

X 

X 

X 

X 

X 

X 

►4 

M 

o 

M 

M 

a 

50 

X 

K 

*4 

X 

*4 

X 

X 

r> 

X 

X 

X 

X 

X 

X 

o 

O 

X 

n 

n 

X 

>  > 
r  r 


> 

r 


1 

n 

(A 

X 

X 

M 

X 

(A 

O 

C 

tA 

M 

X 

tA 

n 

c 

X 

X 

X 

W 

IM 

X 

X 

1 

» 

X 

X 

PI 

3 

X 

X 

P* 

a 

X 

3 

X 

X 

p* 

a 

PI 

X 

pi 

3 

3 

X 

X 

1 

rj 

VI 

r 

r* 

X 

> 

VI 

> 

M 

tA 

X 

> 

tA 

> 

M 

r 

> 

r 

X 

X 

> 

o 

1 

ft 

X 

> 

> 

r* 

a 

X 

tA 

X 

X 

r 

a 

X 

(A 

X 

> 

a 

> 

r 

r 

a 

o 

1 

c r 

PI 

X 

X 

M 

tA 

PI 

tA 

tA 

PI 

M 

tA 

M 

tA 

tA 

X 

tA 

X 

M 

M 

tA 

c 

1 

H- 

3 

w 

M 

n 

X 

3 

n 

3 

o 

X 

3 

n 

M 

X 

M 

n 

o 

X 

n 

1 

►- 

(A 

O 

o 

y 

o 

tA 

tA 

tA 

> 

o 

tA 

tA 

O 

o 

o 

> 

> 

O 

X 

l 

V-' 

a 

a 

X 

X 

X 

X 

a 

X 

a 

X 

X 

X 

tA 

1 

rf 

(A 

(A 

M 

M 

3 

tA 

3 

tA 

w 

w 

3 

1 

H- 

O 

£ 

o 

> 

£ 

O 

o 

y 

\ 

ft 

a 

X 

a 

X 

X 

a 

a 

X 

\ 

W 

tA 

M 

tA 

w 

M 

tA 

tA 

M 

1 

O 

O 

o 

o 

1 

a 

a 

a 

a 

t 

Sr 

ft 

3 

pi 


K> 


175 


Diff  (Musical  Parforianca  Doaiin  -  Expart  Systaas)  15:24  Thursday,  Juna  20,  1991 


ITNO%CFi<7>O>^>0“>(7NUicncnu'<«PUlU*Ul»J»l^i 

vlOV*J<AWKJHOl09)>J9iU1ibwK)HO 


■flCDvjONUI^WKj 


oooooooo 


OOOOOOOOOOOOOOO' 


■«*oooa\oc*OAcr.o<7\OOOrfk0.tkOOOoooa'‘ 


X 

1-4 

H 


> 

r* 


o 

c 


pi 

PI 

PI 

PI 

PJ 

PJ 

PJ 

PJ 

o 

o 

o 

D 

o 

o 

o 

o 

o 

D 

D 

n 

n 

n 

n 

o 

PJ 

n 

o 

1 

» 

< 

< 

< 

< 

< 

< 

<s 

< 

K- t 

t-4 

►4 

>-» 

H 

•—4 

»-» 

*-* 

o 

o 

o 

o 

o 

< 

o 

73 

1 

> 

> 

> 

> 

> 

> 

> 

> 

< 

c 

< 

< 

< 

< 

< 

<; 

< 

< 

< 

ci 

Cl 

ci 

Ci 

a 

> 

Cl 

PJ 

1 

o 

r 

r 

r 

r 

r 

r* 

r 

r 

PI 

PJ 

PJ 

PJ 

PJ 

PI 

PJ 

pi 

PJ 

PJ 

PJ 

z 

z 

z 

z 

z 

f* 

z 

50 

1 

o 

c 

G 

G 

G 

c 

G 

c 

G 

X 

X 

X 

X 

50 

50 

50 

50 

50 

50 

50 

M 

t-4 

>-* 

t-4 

H 

c 

t-4 

> 

1 

B 

> 

> 

> 

> 

> 

> 

> 

> 

c» 

o 

o 

O 

O 

a 

O 

ci 

<n 

a 

ci 

H 

H 

H 

H 

H 

> 

H 

H 

1 

» 

H 

H 

H 

h 

H 

H 

HI 

H 

PJ 

PJ 

PJ 

PJ 

PJ 

pj 

PJ 

pi 

pj 

pj 

pj 

M 

M 

t-t 

►-4 

H 

t-4 

O 

1 

H- 

•-4 

M 

t-4 

1-4 

t-4 

t-4 

H 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

o 

o 

O 

o 

o 

M 

o 

50 

1 

3 

O 

o 

o 

o 

o 

o 

O 

o 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 

z 

z 

z 

z 

z 

O 

z 

z 

Z 

z 

z 

z 

z 

z 

z 

z 

a 

X 

X 

X 

X 

X 

x 

X 

"0 

X 

M 

x 

73 

M 

• 

X 

30 

X 

x 

?0 

50 

X 

50 

50 

50 

50 

"0 

tO 

O 

O 

O 

o 

O 

O 

O 

O 

O 

O 

O 

x 

a 

o 

o 

D 

o 

o  o 

o 

O 

a 

O 

a 

H 

G 

G 

C 

c 

G 

c 

c 

c 

c 

c 

G 

M 

* 

o 

O 

o 

n 

r> 

n 

n 

n 

n 

o 

n 

o 

M 

H 

-1 

H 

H 

h 

H 

H 

h 

H 

H 

H 

• 

*-4 

M 

►4 

t-4 

>-t 

►-» 

t~4 

o 

O 

O 

O 

o 

O 

o 

o 

a 

o 

o 

l 

'-Q 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

1 

00 

> 

> 

> 

OB 

> 

CD 

> 

® 

w 

> 

w 

> 

> 

> 

> 

OB 

> 

> 

> 

> 

> 

> 

</> 

01 

PJ 

G 

G 

G 

PJ 

G 

PJ 

G 

PI 

pj 

G 

PJ 

c 

G 

G 

PJ 

c 

PJ 

c 

G 

G 

G 

c 

G 

PJ 

X 

o 

X 

O 

O 

O 

X 

O 

X 

O 

X 

O 

z 

o 

o 

O 

X 

o 

X 

o 

D 

O 

O 

o 

o 

z 

z 

o 

> 

1-4 

t-4 

> 

1-4 

> 

M 

> 

y 

»-# 

> 

►4 

M 

H4 

> 

> 

►4 

f-4 

►4 

> 

> 

z 

< 

*-i 

H 

HI 

< 

HI 

< 

< 

z 

H 

z 

V| 

HI 

H 

< 

H 

< 

H 

HI 

H 

HI 

H 

H 

z 

z 

H 

>-4 

o 

O 

O 

H 

O 

►f 

O 

M 

H 

O 

H 

o 

O 

O 

M 

O 

O 

O 

O 

O 

o 

O 

H 

H 

PJ 

o 

50 

X 

50 

o 

50 

o 

» 

o 

M 

» 

M 

X 

50 

50 

o 

X 

o 

X 

X 

X 

X 

X 

X 

M 

►4 

z 

50 

*4 

K 

K 

59 

K 

» 

*< 

50 

o 

* 

n 

>< 

K 

K 

X 

K 

X 

»< 

K 

*< 

>< 

*< 

K 

n 

O 

H 

> 

> 

> 

> 

> 

> 

01 

r 

r 

r 

r 

r 

r 

i 

o 

M 

►4 

01 

n 

n 

c 

c 

►4 

H 

M 

H 

H 

01 

X 

n 

n 

c 

G 

M 

H 

01 

n 

c 

t-4 

t-4 

X 

i 

tt 

z 

z 

X 

»< 

r 

r 

a 

a 

a 

s 

z 

X 

X 

*4 

PJ 

r* 

r 

z 

Z 

z 

X 

■< 

r 

a 

z 

z 

X 

i 

tJ 

X 

X 

> 

01 

> 

> 

H 

M 

X 

X 

X 

y 

y 

01 

r 

y 

y 

M 

M 

X 

y 

01 

> 

f-4 

X 

X 

o 

i 

» 

r 

r 

a 

H 

01 

01 

H 

H 

r 

r 

r 

z 

a 

H  > 

01 

01 

H 

H 

r 

z 

H 

01 

H 

r 

r 

a 

i 

cr 

1-4 

►4 

01 

n 

01 

01 

01 

01 

M 

M 

H 

01 

01 

PI 

H 

01 

01 

01 

01 

t-4 

01 

PJ 

01 

01 

M 

►4 

G 

i 

H* 

n 

n 

X 

3 

m 

PI 

o 

O 

n 

X 

X 

Z 

H 

PI 

PI 

O 

X 

Z 

pi 

n 

n 

n 

i 

t— ■ 

> 

> 

o 

01 

01 

01 

y 

y 

y 

o 

o 

01 

o 

01 

01 

> 

o 

01 

01 

y 

> 

H 

i 

H- 

H 

X 

H 

H 

H 

X 

X 

z 

H 

X 

H 

H 

01 

i 

rr 

h4 

H 

z 

M 

►4 

M 

z 

z 

01 

►4 

z 

t-4 

►4 

i 

H- 

o 

o 

> 

o 

o 

o 

y 

y 

O 

> 

o 

o 

i 

• 

z 

a 

H 

z 

a 

a 

H 

Hi 

z 

H 

z 

z 

i 

M 

01 

01 

w 

01 

01 

01 

►4 

M 

01 

w 

01 

01 

i 

O 

o 

O 

O 

i 

a 

z 

a 

Z 

i 

in  in 


176 


Diff  (Musical  Performance  Domain  -  Export  Systams)  15:24  Thursday,  June  20,  1991 


o 


*-  h»  ta 

NJ  M  01 


O 

I 

1 

H» 

03 

i 

1 

u» 

01 

i 

1 

i 

n 

1 

1 

i 

50 

1 

i 

HI 

1 

n 

i 

l-e  ■ 

X 

1 

50 

i 

o  o 

M 

1 

HI 

i 

> 

t 

os 

X 

i 

r 

l 

HI 

i 

1 

> 

i 

l 

r 

I 

1 

i 

**1 

1 

i 

so 

1 

i 

Os  Os 

re 

1 

•a 

i 

o 

1 

so 

i 

C 

1 

NJ 

1*1 

I 

1 

O 

i 

1 

c 

i 

1 

* 

m  pi 

1 

i 

<  < 

O 

1 

i 

>  > 

79 

1 

n 

o 

r  r 

m 

o 

73 

a 

C  c 

50 

o 

ci 

C*J 

M 

>  > 

> 

HI 

2 

50 

X 

X  H 

H 

•*> 

M 

> 

*n 

9-4  9-4 

O 

X 

X 

X 

i* 

O  O 

» 

II 

N 

O 

as 

z  z 

u> 

o 

50 

as 

OS 

A 

1 

1 

01  01 

n 

1 

1 

1 

X  X 

o 

1 

(a 

O 

• 

01  01 

z 

1 

pi 

o 

1 

3  3 

X 

1 

3 

z 

1 

00  0 

n 

1 

> 

X 

1 

o  o 

z 

1 

as 

n 

1 

r  r 

X 

1 

X 

z 

1 

M  f» 

01 

1 

M 

x 

1 

o  n 

1 

o 

01 

1 

1 

1 

1 

1 

1 

HI  X 

1 

t 

1 

M 

1 

3  0 

1 

1 

•a  * 

1 

73 

*0 

1 

r  a 

73 

1 

r* 

50 

1 

HI  01 

0 

1 

M 

O 

1 

n  x 

o 

1 

n 

o 

1 

>  o 

o 

1 

> 

c 

1 

X  » 

c 

1 

H 

n 

1 

M  3 

o 

1 

*~4 

x 

1 

O  > 

X 

1 

o 

oi 

1 

z  X 

01 

1 

as 

1 

01  H» 

1 

01 

1 

o 

1 

Z 

( A 


r 


w 


O 

i 

If 

M 

a 

i 

M- 

o 

CD 

PS 

us 

D  U  Kl 

9* 

01 

i 

3 

i 

i 

o 

i 

r% 

n 

i 

50 

i 

PI 

M 

i 

►— 

M 

N- 

h- 

X 

i 

• 

o 

o 

o 

o 

o 

CD 

Os  Os  Os 

A 

HI 

i 

■ 

> 

i 

• 

r 

i 

3 

i 

if 

i 

(A 

i 

c 

x 

i 

=r 

50 

i 

• 

as 

OB 

CD 

Oft 

as 

as 

A  M  A 

K> 

PI 

i 

n 

o 

i 

• 

G 

i 

i 

O 

i 

o 

i 

■ 

i 

» 

PI 

m 

w 

PI 

n 

n 

pi  n  n 

PI 

t 

H* 

< 

< 

< 

< 

o 

o 

<  o  o 

< 

O 

i 

3 

> 

> 

> 

> 

o 

o 

»  a  o 

> 

73 

r 

r 

r 

r 

z 

z 

r  z  z 

n 

o 

9 

G 

c 

c 

c 

w 

M 

G  M  M 

c 

9 

M 

• 

> 

> 

> 

>  X 

H 

>  X  X 

> 

> 

X 

•a 

M 

H* 

4  H  H 

X 

X 

» 

M 

M 

M 

M 

o 

O 

MOO 

M 

o 

M 

o 

o 

o 

O 

z 

z 

o  z  z 

o 

50 

Ul 

• 

z 

z 

z 

z 

z 

z 

e 

n 

W 

• 

1 

UJ 

I 

»7 

< 

< 

< 

<5 

X 

< 

01  O!  01 

01 

n 

♦ 

• 

M 

M 

M 

M 

w 

HI 

pi  n  pi 

PI 

o 

1 

m 

01 

01 

01 

01 

01 

01 

3  3  3  3 

z 

1 

rr 

G 

c 

G 

G 

G 

G 

>  >  > 

> 

X 

1 

• 

> 

> 

> 

> 

> 

> 

z  z  z 

z 

PI 

1 

*7 

r 

r 

r 

r 

r 

r 

XXX 

X 

z 

1 

M  H  M 

HI 

X 

1 

rr 

non 

n 

01 

1 

ST 

1 

• 

1 

| 

a 

1 

pi 

X 

0) 

50 

n 

9 

01 

50  X  50 

M 

73 

1 

50 

X 

PI 

r 

PI 

X 

PI  50  PI 

3 

9 

1 

> 

01 

r 

> 

r 

01 

r  >  r 

9 

o 

1 

01 

z 

X 

► 

01 

> 

X 

>  z  > 

r 

o 

1 

• 

01 

PJ 

H 

01  X 

PI 

X  01  X 

M 

c 

1 

X 

3 

M 

m 

M 

z 

M  M 

o 

n 

1 

n 

o 

01 

o 

01 

o 

01 

O  O  O 

> 

X 

1 

• 

50 

z 

z 

Z  50  z 

X 

01 

1 

•o 

3 

01 

01 

01  3  01 

M 

1 

m 

> 

> 

O 

I 

cr 

H 

X 

z 

1 

H- 

HI 

w 

01 

1 

M 

o 

O 

1 

H- 

z 

z 

1 

rr 

01 

01 

1 

H- 

» 


177 


Diff  (Troubleshooting  Doaein  -  Expert  Systems)  14:57  Thursday,  June  20,  1991 


I 

R 


D.  CURRENT  DOMAIN  EVALUATION  TOOLS 


180 


QuciuMnbatj 
A&qouip*!  p«uiA()y  [ 


aijjojd  6u!U88jos  uoijBonddv  SMI  :  1*2  9JnB|j 


s 


181 


APPLICATION  SCREENING  PROFILE 


RISK  ASSESSMENT 


Project  Title: 


bs<r  Organization: 


L'ser  POCfsj: 


The  objective  here  is  to  measure  the  amount  of  risk  associated  with  the  successful 
development  and  implementation  of  an  Expert  System  for  use  by  a  requesting  user 
organization. 

A  minimum  of  point  (out  of  the  maximum  )  can  be  generally  considered 

sufficiently  risk  free  for  development  within  the  AI  office.  Some  of  the  risks  can  be 
diminished  through  discussion  with  user  management  or  the  purchase  of  other 
hardware/software.  Other  risks  can  not  be  overcome. 

Each  category  has  a  maximum  score  associated  with  it  Within  each  category  (or  sub¬ 
category)  questions  have  a  maximum  score  associated  with  them.  These  are  guideline 
only.  The  maximum  points  for  a  group  of  questions  can  be  redistributed  among  those 
questions  as  may  be  required  under  some  circumstances.  Likewise,  the  maximum  points 
for  a  group  of  categories  may  be  redistributed.  Care  and  forethought  should  be  given 
to  the  redistribution  of  any  points. 


TOTAL  POINTS  (out  of  ): 
RISK  (high,  medium,  low): 
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MANAGEMENT  RISK 


Solution  Awareness 

//  manapemem  ls  not  aware  o'  the  solution  means  and  die  capabilities  /  timuanons  of  dial  solution  means,  they 
mav  be  unwdlmy  to  support  a  in  me  future 

A.  Does  the  user  organization  already  own  hardware  and/or  software  for 
Expert  System  technology? 

ii  :  4 


No  Some  Yes 


B.  Does  the  user  organization  have  realistic  expectations  of  Artificial 
Intelligence  and  Expert  System  technology? 

(13  6 


No  Some  Yes 


C.  Is  the  user  management  aware  of  the  solution  being  considered? 

2  4 

4 - 1 

No  Some  Yes 

D.  Is  the  user  management  skeptical  of  Artificial  Intelligence  or  Expert 
System  technology? 

0  3  6 


Yes 


Some 


No 


Total 
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MANAGEMENT  RISK  (cont) 

Cost  Awareness 

If  user  management  is  not  aware  of,  anti  willing  to  support  the  cost  of  development,  the\'  ma\-  be  unwilling  to  funa  those  costs 
m  the  future 

A.  Are  the  project  assumptions,  costs,  and  constraints  understood  and 

accepted  by  the  user  management?  _ 


0 

1 

10 

I 

20 

I 

No 

Some 

Yes 

R.  Has  the  organization  ever 

developed  any 

AI  system? 

0 

1 

5 

} 

10 

1 

No 

Some 

Yes 

C.  Is  it  expensive  or  time 
problem? 

consuming  to 

train  others  to  solve  the 

0 

( 

5 

-  1 

10 

1 

No 

Somewhat 

Yes 

Total 
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MANAGEMENT  RISK  (cont) 

Value  Awareness 

If  user  management  is  not  aware  of  the  potential  benefits  available  to  them,  they  may  be  unwilling  to  support  the  effort  now  or 
m  the  future. 


A.  Is  improved  understanding  of  the  current  problem-solving  process 
viewed  as  valuable  to  your  organization? 

(I  I  2 

j - 1 - 1 

No  Some  Yes 

B.  Is  solving  this  problem  a  priority  issue  for  the  user  organization? 

0  3  6 

| - 1 - 1 

No  Somewhat  Yes 

C.  Is  the  expert  required  to  spend  excessive  amounts  of  time  helping 
others? 


0  2  4 

1 - 1 - 1 

No  Some  Yes 

D.  Are  excessive  amounts  of  some  resources  needed  that  could  be  used 
more  effectively  or  used  elsewhere  if  this  problem  did  not  exist? 

0  2  4 

| - 1 - 1 

No  Somewhat  Yes 

Total 


185 


MANAGEMENT  RISK  (cont) 

Overall  Management  Support 

This  is  just  an  overall,  across-the-board  assessment  of  support 

A.  Is  user  management  committed  to  solving  the  problem  with  Expert 
System  technology? 


0  5  10 


Not  Somewhat  Yes 

committed  committed  committed 

B.  Do  several  layers  of  user  management  support  this  development  effort? 


0  5  10 


No  Some  Yes 


Total 


186 


USER  AWARENESS  /  APPROVAL  /  SUPPORT 

If  the  user  wtU  not  support  or  use  the  system,  it  will  be  a  failure  no  matter  how  wonder  ful  the  system  rntry 
A.  Do  the  users  view  the  solution  with  a  non-hostile  attitude? 

0  7  1/2  IS 


Yes  .  Some  No 

hostility  hostility  doesn’t 

exists  exists  exists 

1$.  Will  the  users  actually  use  the  proposed  system? 

0  10  20 


No  Some  Yes 

C.  Are  there  realistic  expectations  of  the  proposed  system? 

0  21/2  5 

| - 1 - 1 

No  Some  Yes 

D.  Do  the  users  perceive  any  problems  with  ES  approach? 

0  5  10 


Many  Some  No 

problems  problems  problems 

E.  Do  the  users  fear  or  hate  the  proposed  system? 

0  5  10 


Yes 


Some 


No 


Total 


187 


TECHNOLOGY  RISK 
Appropriate  To  ES 

The  solution  must  be  " do-able "  efficiently  and  effectively  tvtth  Expert  System  technology'. 

A.  Are  heuristics,  techniques,  or  methods  involved  in  solving  the  problem? 


0  15  30 


No  Some  Many 

B.  Would  an  algorithm  or  a  conventional  programming  solution  be  too 
difficult  or  time  consuming  to  attempt? 


0  7  1/2  15 


Sot  Somewhat  Yes 

C.  Does  the  solution  involve  a  relatively  constrained  solution  space? 


0  15  30 


No  Somewhat  Yes 


D.  Have  conventional  solutions  been  ruled  out? 

0  7  1/2  15 


No  Somewhat  Yes 


Total 


188 


TECHNOLOGY  RTSK  (cont) 

Possible  With  ES 

Sot  ail  systems  will  be  possible  to  develop  with  Expert  System  technolog.'. 
A.  Is  the  problem  solution  generally  accepted/agreed  upon? 


Not 

agreed  upon 


Somewhut 
agreed  upon 


Yes 

agreed  upon 


It.  Is  the  problem  area  basically  simple  or  can  it  be  broken  into  simple 
independent  sub-problems? 


Complex 


Averuge 


Simple 


C.  Does  the  problem  solution  depend  upon  a  narrowiy  defined  body  of 
knowledge  excluding  common  sense. 


Somewhat 


D.  Is  there  a  recognized  source  of  available  and  reliable  knowledge? 


E.  Can  the  expert  articulate  the  problem  solving  process? 


Somewhat 


F.  Has  the  expert  ever  trained  someone  else  to  solve  this  problem? 


G.  Is  the  expert  willing  to  commit  to  the  development  effort? 
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No 


Some 


Yes 


Total 


190 


TECHNOLOGY  RISK  (cont) 

Justification  Of  ES 

There  must  be  a  real  value  associated  with  and  sufficient  to  justify  resource  expenditures  to  develop  a  system.  If  there 
is  not  sufficient  justification,  development  support  will  be  weak. 

A.  Is  there  a  need  for  consistent  application  of  policy  and  procedures  that  _ 

are  not  achieved  today  or  are  achieved  only  at  great  cost? 


0  3  6 


No  Some  Yes 


B.  Is  there  a  possibility  of  losing  the  knowledge? 

0  3  6 


No  Some  Yes 


C.  Is  there  a  need  to  distribute  this  knowledge  to  others? 
0  3  6 


No  Some  Yes 


D.  Does  the  problem  solution  require  scarce  resources  or  knowledge? 
0  3  6 


No  Some  Yes 


E.  Is  an  expert  repeatedly  required  to  help  others  solve  this  problem? 
0  3  6 


No  Sometimes  Yes 


Total 


191 


PERFORMANCE  RISK 
Accuracy  /  Uncertainty 

Can  the  accuracy  required  be  provided  by  Expert  Syrian  technology. 
A.  Does  the  problem  involve  uncertainty? 


0  41/2  9 


Yes,  Some  No 

uncertainty  uncertainty  uncertainty 

exists  exists  exists 

H.  If  uncertainty  is  involved,  can  less  than  complete  accuracy  be 
tolerated? 

0  41/2  9 


No  Somewhat  Yes 


Total 

Speed 

Is  it  possible  to  reach  the  response  tune  required;  real  time. 

A.  What  are  the  required  response  times/rates? 

0  12  3 


Millisecond  Second  Minutes  No  requirement 

Note:  Response  time/rate  requirements  can  limit  processing  ability. 

C.  Can  the  hardware  /  software  /  communication  requirements  and  costs 
be  met  or  procured  to  meet  the  response  time? 


0  1  3 


No  Somewhat  Yes 


Total 


192 


PERFORMANCE  RISK  (cont) 

Interfacing 

If  the  required  interface  can  not  be  done,  the  information  wiU  have  to  be  retrieved  and  entered  manually  Thu  will 
lower  speed  and  efficiency  as  well  as  increasing  the  chances  for  error.  As  thts  becomes  more  of  a  problem,  the 
likelihood  of  system  usage  diminishes 

A.  Is  there  frequent  interaction  among  system  modules(Files)?  _ 


0  I 


High  interaction  Moderate  Interaction  Low  interaction 

umong  modules  among  modules  among  modules 

B.  Are  the  interfacing  systems  classified? 


Yes  No 


C.  Is  the  hardware  and  software  plutform  of  the  ES  compatible  with 
interfacing  hardware  and  software? 


1  2  5 


No  Somewhat  Yes 


Total 


193 


KNOWLEDGE  ENGINEERING  RISK 

A.  Can  the  expert  express  the  knowledge  (solution  method)  verbally? 
(telephone  test) 


Extremely  difTicull 
to  verbalize; 
performance  based 
expression. 


Partially  expressible 
verbally;  requires 
some  performance. 


Dimculi  io 

express 

verballv. 


express 

verbally. 


n.  Is  the  domain  expertise  available. 


No  expertise 
available. 


Only  analytical 
expert!'  ?. 


Some  operational 
expertise. 


Significant  operational 
expertise. 


C.  Are  there  representative  cases/scenarios  of  the  problem  available? 


rseudo  case/scenarios 
■re  difficult  la  make. 


Pseudo  case/ scenarios 
can  be  easily  made. 


Adequate  representative 
coverage. 


D.  Does  the  solution  have  a  high  degree  of  agreement  among  the  experts? 


Lillie  agreement. 


Moderate  agreement. 


Highly  agreed  upon. 


EL  How  many  experts  are  needed  to  adequately  cover  the  domain? 


Domain  can  not 
be  adequately 
covered. 


Experts  specialize 
but  do  not  know 
about  Ibe  entire  domain. 


Experts  specialize 
but  some  know  abo"< 
the  entire  domain 


F.  How  often  does  the  knowledge  and/or  the  process  to  solve  the  problem 
change? 


More  than  once  a 
month.  (System 
maintenance  will 
exceed  system 
development  effort) 


Once  a  month. 


Once  a  year. 
(System  will 
require  moderate 
maintenance) 


Lillie  If  ever. 
(Once  built, 
system  will 
require  little 
maintenance) 


194 


KNOWLEDGE  ENGINEERING  RISK  (cont) 

G.  is  the  knowledge  straightforward  (can  it  be  represented)? 


NO  Some 

Very  t^ifllculL  Moderately  difficult;  convoluled. 


Yes 

Not  Difficult. 


II.  llow  large  do  you  estimate  the  final  knowledge  hase(s)  to  be? 


More  Ihnn  1300 
elementjw'ohjecu. 


700  .  1300 
e  le  me  n  tsJobjec  is. 


200  .  700 
elementVobJects. 


Less  limn  200 
clemenls/objecLs. 


1.  Can  a  prototype  be  accomplished  over  n  subset  of  the  problem  domain 
to  udequaudv  prov<*  the  worth  of  such  a  system? 


A  meaningful  prototype 
■aa  not  be  bullC 


A  very  lurge  prototype 
would  have  to  be  built 


Meaningful  prototype 
can  be  done. 


J.  Are  there  commercially  available  tools  that  can  be  used  to  develop  this 
svstem? 


Tools  are  not  available; 
loo1'  must  be  developed. 


Tools  arc  at  the  Util 
end  of  research. 


Tools  are  available 
commercially. 


K.  Is  the  required  technology  mature  enough  to  rely  on? 


Somewhat 
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SYSTEM  ENGINEERING  RISK 

A.  Based  on  the  number  of  separate  modules,  how  complex  is  the  system? 
1  3 


Several  expert  systems  Single  expert  system 

and  conventional  modules-  module  and  one  or  more 

conventional  modules. 


B.  Is  there  a  high  degree  of  interaction  between  modules? 
I  4 


Single  expert  system 
module. 


7 


High  Interaction.  Moderate  Interaction.  Low  Interaction. 

C.  VVhat  response  time  is  required? 

13  4  5 


Milliseconds.  A  few  seconds.  A  few  minutes.  No  requirement 


D.  Would  the  requirements  on  hardwnre(both  development  &  delivery)  be 
readily  available? 

0  2  4 


No  Somewhat  Yes 

(Very  restricted)  (Unrestricted) 

E.  Does  the  delivery  environmental  ready  exist  in  the  field,  and  is  it  a 
standard  configuration? 

2  1/2  5 


No 


Somewhat 


Yes 


F.  Are  there  any  restrictions  on  what  software(both  development  & 
delivery)  choice  can  be  made? 

0  2  1/2  5 


Yes  Some 

(Very  restricted;  (Somewhat 

few  tools  available)  restricted;  some 

toots  excluded) 


No 

(Unrestricted) 


G.  Will  the  prototype  system  scale  to  a  full  system  without  major 
redesign? 


196 


0 


2 


10 


Redesign  required  lo 
Increase  scope  and 
functionality 


No  redesign  required, 
but  substantial 
augmentation  required 
to  increase  scope  and 
functionality 


No  redesign  required 
and  omv  moderate 
augmentation  to 
Increase  scope  and 
functionality 


197 


SYSTEM  ENGINEERING  RISK  (cont) 


II.  How  quickly  does  the  solution  and/or  the  problem  domain  change;  can 
we  expect  it  to  change  during  the  development  period? 

0  21/2  5 


Changes  frequently  Likely  to  change  Not  often 

during  development 

I.  Can  the  prototype  knowledge  base  be  used  in  full  scale  development 
or  will  the  knowledge  have  to  be  recoded/designed/structured? 

0  2  4 


Must  be  re-deslgn/  Will  have  lo  be  Used  with  little 

re-ctructure/re-code  recoded  Modification 


J.  IIow  changeable  will  the  data,  models,  and  modules  comprising  the 
system  be  over  time? 

0  5  20 


System  maintenance 
will  exceed  system 
development  efTort 


Moderate  maintenance 


Little  maintenance 
after  built 


Total 


196 


ECONOMICAL  RISK 


A.  Does  the  life  cycle  cost  of  the  system  exceed  the  payoff.1 


Layoff  much  less  than 

lile  cvcle  cost  of 

'  • 

the  svsiem 


Moderate  payoff 


I’livoff  much  greater 
than  life  cycle  cosl 
of  the  svstem 


B.  Could  the  basic  structure/framework/knowledge  of  the  system  be 
applied  to  other  applications  within  the  organization? 


Low  generally, 
useful  for  this 
application  only 


Moderate  generally 


High  generally; 
adaptable  lo  many 
high  payoff  applications 


C.  What  is  the  estimated  investment  in  time,  money,  equipment,  etc.  to 
design,  build,  test  and  Held  the  system? 


Moderate 
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APPENDIX  B  ~  TAROT  METRIC 


3.1  INTRODUCTION 

The  system  chart  of  ESEM,  Figure  1,  shews  the  foilwing  three  risk-based 
decision  points: 

1.  At  project  initiation.  -  the  KE  must  make  a  determination,  first, 
on  the  suitability  of  ES  technology  for  the  project,  and, 
second,  on  the  risk  of  developing  and  using  the  automated 
system. 

2.  At  the  completion  of  each  stage  of  work.  -  the  KE  mist  estimate 
the  reduction  in  risk  achieved  in  the  stage. 

3.  Aft-m-  stooping  stage  work.  -  the  KE  must  assess  the  risk  in 
moving  the  project  to  the  conventional  development  methodology. 

To  assist  in  the  evaluation  of  a  project  for  ES  suitability  and  in 
estimating  risk,  the  factors  listed  in  Table  B-l  should  be  used.  The  table 
uses  levels  of  indention  to  indicate  factors  that  roll  into  less  indented 
factors  above  them.  The  overall  scare  for  suitability  is  the  TAROT  metric 
value;  it  depends  on  the  values  of  all  of  the  factors  in  the  table.  It  is 
not  assigned  independently;  instead,  it  is  determined  by  the  the  values  of 
all  the  other  factors. 

Projects  vary  widely  in  their  characteristics,  and  not  all  the  factors 
listed  in  Table  B-l  are  necessarily  relevant  far  a  given  project.  Also, 
other  factors  may  be  added  to  the  list  far  other  projects.  This  metric  is 
primarily  geared  towards  larger  sized  projects  and  should  be  adapted  for 
projects  smaller  in  size.  Also,  even  though  a  project  has  been  assigned  as 
an  ES  project,  a  suitability  analysis  should  still  be  performed  to  ircsira 
that  nothing  was  missed  when  making  the  initial  assessment,  and  that  this  is 
a  suitable  candidate  for  an  ES  implementation  base  an  the  information 
available. 

The  factors  in  the  table  can  be  viewed  sinply  as  a  checklist  of  items  that 
should  be  reviewed  before  making  any  decision  or  can  be  viewed  in  a  more 
technical  sense  as  the  factors  in  a  decision  rule.  When  viewed  as  a  set  of 
factors  with  a  set  of  rules  for  combining  the  factors  in  an  automated  rules- 
based  decision  system,  they  form  part  of  an  ES.  The  use  of  an  ES  to 
evaluate  the  suitability  of  a  project  for  using  ES  technology  is  not 
recommended  because  the  wide  variation  in  projects  would  require  developing 
a  new  rules  base  for  each  possible  application. 

The  factors  can  be  used  more  directly  as  part  of  a  weighted  decision  rule. 
This  approach  to  decision-making  is  more  flexible  than  the  rules -based 
approach  and  has  often  been  used  for  choosing  among  possible  hardware  suites 
and  for  "make  or  buy"  decisions. 
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TtbU  B-l. 


Evaluation  f  actors  (or  ES  Dndopsnt 


factors 


POSSIBLE  VALUE  RATINGS 


TAROT  metric 
(overall  suitability) 
Worth 

Payoff /Cost 

Target  functionality 

Priority 


Poor,  Fair,  Good 

Negative,  Lott,  Moderate,  High 

<  t.  1  •  1.5,  t.5  •  3,  >  3 

<  5 OX,  50X-73X,  75X-90X,  >  90X 

Useful,  Necessary,  Crucial 


Expertise  Needed  in: 


Many  Locations 

No, 

Tes 

Hostile  Envirorawnt 
Location  Difficult  to 

NO, 

Tea 

Access 

Expertise  Skills: 

NO, 

Tes 

Being  Lost 

NO. 

Tes 

Scarce 

NO, 

Tes 

Risk 

Coopt  exity 

I ntui  t i on/ Common  Seme 
Technology 
Decision  Definition 
Knowledge  Oeapin 
Experts 
Control 
Site 

Autorwaay 

X  Conventional 
Interface  Requi Tenants 
Eoptoyee  Acceptance 
Solution  Available 
Easier  Solution 
T  eachabi l i ty 


Low,  Moderate,  High 
Low,  Moderate,  High 

<  10X.  I OX  •  5 OX,  >  SOX 
Build,  Enhance,  Modify,  Exists 
Fust.,  OK,  Well-Defined 
Eclectic,  OK,  Marrow 
Available,  OK,  Unavailable 
Tight,  OK,  Looae 

Saall,  Modi  us,  Large,  Very  Large 
Advisor,  Aide,  Assistant,  Agent, 
Adainistrstor  (Supervisor) 

»  5 OX,  10X  -  SOX,  «  10X 
user.  File,  Coen,  Pil ,  Message 
Negative,  Neutral,  High 
Adequate,  Partial,  Hone 
Complete,  Partial,  None 
Difficult,  Possible,  High 
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B . 2  USE  CF  FACTORS  IN  A  DECISION  RULE 

To  use  the  factors  in  a  decision  rule,  numerical  values  should  be 
to  each  of  the  possible  values  for  each  independent  factor;  the  independent 
factors  are  those  that  do  not  have  an  indented  set  of  subfactors.  For 
example,  the  values  of  the  Solution  Available  factor  may  be  assigned  as 
follows: 

o  Adequate:  1 
o  Partial:  5 
o  None:  10 

To  aid  in  interpreting  the  factors,  Table  B-2  lists  then  by  origin. 

Next,  the  actual  values  for  the  proposed  project  rust  be  evaluated.  As  an 
example,  it  may  be  known  that  there  is  no  solution  available  to  the  problem 
for  which  the  system  is  intended  to  solve,  so  the  attribution  of  this 
factor  would  be  10  for  the  proposed  project. 

Next,  assign  a  weighting  far  the  importance  of  each  factor.  Far  example,  an 
a  project  of  high  priority,  the  possible  acceptance  of  the  intended  system 
by  employees  (the  Errployee  Acceptance  factor)  may  be  considered  to  be  of 
little  importance  and,  thus,  be  assigned  a  lav  weight. 

A  linear  decision  function  can  be  constructed  by  farming  a  weighted  sum  of 
factor  numerical  values.  The  decision  is  then  made  to  proceed  if  the 
weighted  sum  is  above  a  certain  threshold,  and  not  to  proceed  if  the 
decision  is  below  the  threshold.  The  value  of  the  threshold  is  not  set 
arbitrarily;  it  is  evaluated  by  assuming  extreme  cases  far  the  set  of 
relevant  factors  and  than  judging  what  would  be  a  prudent  value. 

Use  of  the  table  yields  values  far  three  key  factors:  suitability  (the  value 
for  the  TAROT  metric) ,  risk,  and  worth.  The  numerical  values  for  risk  and 
worth  (the  assignment  of  numbers  to  the  ratings  of  poor,  fair,  and  good)  axe 
a  spinoff  from  the  confutation  of  suitability.  Breakpoints  axe  set  far 
these  subsidiary  factors  in  the  same  way  as  far  suitability. 

There  are  two  principal  criteria  far  linear  decision  functions:  they  must  be 
robust  and  they  should  exhibit  sensitivity  to  critical  variables. 

Robustness  implies  that  small  changes  in  the  input  variables  should  not 
cause  wide  swings  in  the  dependent  variables  or  cause  the  decision  to  go  the 
other  way.  Sensitivity  implies  that  a  shift  in  a  key  variable  should  cause 
the  decision  to  shift.  These  two  criteria  are  to  seme  extent  conflicting, 
and  experimentation  with  weights  may  be  needed  to  find  the  decision  function 
most  suitable  for  a  given  project. 
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T»bU  B-l.  Factors  in  ES  Development  by  Oricin 


f^cl9*S  VALUES 


Problem  factor*: 

Solution  Available 
Easier  Solution 
Complexity 

I  ntui  t  i  on/Coonnn  Same 
Technology 
Oaciaion  Definition 
Knowledge  Oomein 

Object  Syataa  factors: 


Adequate,  Partial,  Nona 
Mona,  Partial,  Coopiete 
Lou,  Moderate,  High 

«  10X,  10*  •  50*,  >  50* 

Build,  Enhance,  Modify,  Exists 
Fuzzy,  OK,  Uell-Oefined 
Eclectic,  OK,  Narrow 


Size 

No,  Rules 

Type  of  Inference 

NU 

KA  Module 
Autonomy 

interface  Requirements 
Z  Conventional 
Expertise  Needed  in: 

Many  Locations 
Hostile  Environment 
Location  Difficult  to 
Access 

Development  System  factors: 

Hoet  Computer 
Development  Environment 
Languages,  Shells,  and  Tools 

Environmental  Factors: 

Expertise  Skills: 

Being  Lost 
Scarce 

Ensiloyee  Acceptance 
Teachability 
Level  of  Staff  Skill 
Experts 

Mo.  of  Experts 
Expert  Availability 
Turnover 
Expert  Attitude 

Control 

Oats  Control 
Procedure  Control 
Performance  Metric 


Smalt,  Medius,  Large,  Very  targe 

<  50,  50*200,  200*500,  500*1000,  »  1000 
Production,  Resolution,  Analogy, 

Generate  end  Test.  Statistical 
mo.  Tea 
no.  Yes 

Advisor,  Aide,  Assistant,  Agent,  Adeinistrator 
User,  Fite,  Com,  PL!,  Message,  Multiple 
»  50*,  10*  -  50*,  <  10* 

NO,  Yes 
NO.  Yes 

No,  Yes 


Limited,  OK.  Powerful 
Poor,  OK,  Rich 
Few,  OK,  Many 


no.  Yes 
no,  Yes 

Negative,  Neutral,  High 
Difficult,  Possible,  High 
0*1  year,  1*3  years,  3*5  years,  »  5  years 
Unavei table,  OK,  Available 
0,  1  •  3,  »  3 

<  50*,  5QX-75X,  75X-90X,  »  90* 

Low,  Moderate,  High 
Hostite,  Inarticulate, 

Uninterested,  Milling 
loose,  OK,  Tight 

Low,  Moderate,  High 
Low,  Moderate,  High 
Low,  Moderate,  High 
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3.3  DISCUSSION  OF  FACTORS 

The  key  factors  In  a  decision  rule  are  diaficribed  below. 

Suitability.  Ibis  factor  provides  a  measure  of  the  overall  suitability  of  a 
project  for  possible  implementation  as  an  ES.  It  is  a  dependent  factor 
which  must  be  found  by  evaluating  the  independent  factors  and  then  combining’ 
them  in  weighted  fashion. 

Risk.  This  dependent  factor  is  evaluated  before,  during,  and  after  the 
project.  It  provides  an  index  far  monitoring  progress  and  rrrrt-riHTt-trrj  to 
the  initial  estimate  of  suitability. 

worth.  This  factor  is  concerned  with  the  potential  benefit  that  might  be 
derived  from  a  successful  system  development.  It  is  primarily  evaluated 
only  at  the  start  of  ESEM,  but  it  might  be  reevaluated  if  knowledge  acquired 
during  the  project  refines  estimates  of  the  factors  oryrt-HHTr-irirj  to  worth. 

Employee  Acceptance.  In  general,  if  the  user  base  intended  for  a  system  is 
opposed  to  using  it  (conventional  or  expert) ,  then  the  chances  are  high  that 
the  system  may  not  be  used  or  may  not  be  used  to  its  capacity.  A  low 
employee  acceptance  for  an  ES  has  a  negative  impact  on  the  suitability  far 
that  system. 

Solution  Available.  If  an  automated  solution  already  exists  for  a  given 
problem,  then  the  possible  gains  to  be  had  from  developing  a  new  one,  baaed 
on  the  use  of  ESs,  may  be  small.  Indirectly,  this  affects  both  Suitability 
and  Worth. 


SaaifiE  solution.  If  a  solution  not  only  already  exists,  but  is  easier  to 
implement  than  an  ES,  then  the  use  of  an  ES  approach  to  the  problem  must  be 
strongly  questioned. 


\  j«ty-  a  manual  system  exists  to  aoccxqplish  a  certain  task,  and 
if  the  system  is  easily  taught  to  operators,  then  the  likelihood  is 
increased  that  an  ES  can  easily  be  developed  to  acoanplish  the  task. 

However,  if  use  of  the  manual  system  is  not  easily  taught,  then  it  is  less 
likely  that  a  satisfactory  ES  can  be  developed  to  perform  the  task. 

Pavoff /Cost.  This  factor  is  the  key  factor  in  the  evaluation  of  worth.  The 
cost/benefit  analysis  is  so  well  known  and  widely  used  far  systems  of  all 
types  that  it  needs  no  explanation  here.  However,  it  should  be  noted  that 
the  payoff /cost  factor  is  only  one  of  five  factors  that  should  be  considered 
in  evaluating  the  worth  of  a  possible  project. 


Target  Functionality.  If  the  target  functionality  for  a  proposed  system 
does  not  new  exist  or  cannot  be  accomplished  in  any  other  way,  the  possible 
importance  of  this  factor  may  be  sufficiently  high  that  it  could  outweigh 
other  factors. 
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Priority-  The  priority  of  a  proposed  system  has  an  obvious  bearing  on  the 
worth  of  the  project. 

Expertise  Needed,  if  expertise  of  a  special  nature  is  needed  in  many 
locations,  in  a  hostile  environment  or  in  locations  that  are  difficult  to 
access,  the  potential  worth  of  an  ES  mild  be  increased. 

Expertise  Skills.  If  the  skills  of  experts  are  being  lost  due  to  attrition 
of  personnel,  or  if  they  are  scarce,  then  the  worth  of  a  proposed 

ES  to  duplicate  these  functions  nay  outweigh  other  factors,  including  a 
high-risk  factor.  ESs  have  been  proposed  far  many  applications  simply 
because  there  is  a  desire  to  capture  the  expertise  of  humans,  and  the  ES  is 
a  very  effective  tool  far  this  purpose. 

Cgnpjggisy.  In  general,  the  higher  the  complexity  of  a  wanna i  system,  the 
less  chance  that  it  can  be  duplicated  by  an  ES  (i.e. ,  the  greater  the  risk 
of  development  for  the  project) .  This  factor  is  related  to  the  1  < fry 

of  manual  systems,  but  can  be  evaluated  independently  when  there  is  no 
on  the  teachability  of  the  system. 

Experts.  At  present,  there  is  a  high  demand  for  human  experts  frtxa  whan  the 
expertise  for  a  given  task  can  be  extracted,  The  presence  of  a  sizable 
group  of  experts  to  serve  as  part  of  the  t-gjww  far  the  proposed  project 
reduces  the  risk  of  development.  The  further  development  of  am-rmw+wi 
tools  for  knowledge  acquisition  may  reduce  the  importance  of  this  in 

the  future. 


Control.  The  term  control  refers  here  to  that  which  is  exerted  in  the 
manual  system  to  be  duplicated  by  machine.  In  general,  a  system  that 

is  well  controlled  (even  though  crap  lex)  is  ««iw  to  rftipi  irara  than  one 
that  is  poorly  controlled.  This  factor  has  a  direct  bearing  on  risk. 

Size.  The  rule  of  thumb  in  ESs  is  as  follows:  there  is  no  task  too 
for  an  ES.  In  general,  the  larger  the  size  of  the  expected  system,  the 
greater  the  risk  of  attenpting  to  develop  an  ES  to  duplicate  the  function. 

Autonomy.  If  a  proposed  system  is  required  to  function  with  a  high  degree 
of  autonomy  (e.g. ,  working  far  long  periods  of  •Hma  without  human 
supervision  or  making  critical  decisions) ,  the  risk  of  development  for  that 
system  is  high.  The  additional  testing  and  special  techniques  that  must  be 
employed  to  ensure  robustness  for  an  autonomous  system  can  add  significantly 
to  the  cost  of  the  system  and  to  the  risk  in  undertaking  development. 

Percent  Conventional.  In  general,  a  system  with  a  high  percentage  of 
conventional  software  ccnponents  and  only  a  small  percentage  of  cules-based 
or  other  ES  elements  is  less  risky  than  one  that  has  a  high  percentage  of  ES 
elements.  However,  this  factor  must  be  tempered  by  the  total  size  of  the  ES 
uonponent  required. 

TntrT*faQt»  Requirements.  An  ES  that  interfaces  only  with  a  human  being, 
providing  advice  in  response  to  a  dialogue  by  the  user  is  generally  a  lower 
risk  system  than  one  that  rust  interface  with  other  systems  in  addition  to 
the  user. 
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